{ "source_file": "0930", "source_platform": "个人博客", "author": "Jay Alammar", "title_listed": "The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)", "keywords": "BERT,ELMo,迁移学习", "domain": "文本处理", "content_type": "图解教程", "is_paid_flag": "否", "url": "https://jalammar.github.io/illustrated-bert/", "publish_date": "2018-12", "corpus_stage": "预训练/SFT", "notes": "BERT图解，NLP迁移学习经典", "id": "84b0f50c3cc1", "collected_at": "2026-10-02T14:18:03", "title": "The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.", "body_markdown": "The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)\nDiscussions:\nHacker News (98 points, 19 comments)\n,\nReddit r/MachineLearning (164 points, 20 comments)\nTranslations:\nChinese (Simplified)\n,\nFrench 1\n,\nFrench 2\n,\nJapanese\n,\nKorean\n,\nPersian\n,\nRussian\n,\nSpanish\n2021 Update:\nI created this brief and highly accessible video intro to BERT\nThe year 2018 has been an inflection point for machine learning models handling text (or more accurately, Natural Language Processing or NLP for short). Our conceptual understanding of how best to represent words and sentences in a way that best captures underlying meanings and relationships is rapidly evolving. Moreover, the NLP community has been putting forward incredibly powerful components that you can freely download and use in your own models and pipelines\n(It’s been referred to as\nNLP’s ImageNet moment\n, referencing how years ago similar developments accelerated the development of machine learning in Computer Vision tasks)\n.\n(ULM-FiT has nothing to do with Cookie Monster. But I couldn’t think of anything else..)\nOne of the latest milestones in this development is the\nrelease\nof\nBERT\n, an event\ndescribed\nas marking the beginning of a new era in NLP. BERT is a model that broke several records for how well models can handle language-based tasks. Soon after the release of the paper describing the model, the team also open-sourced the code of the model, and made available for download versions of the model that were already pre-trained on massive datasets. This is a momentous development since it enables anyone building a machine learning model involving language processing to use this powerhouse as a readily-available component – saving the time, energy, knowledge, and resources that would have gone to training a language-processing model from scratch.\nThe two steps of how BERT is developed. You can download the model pre-trained in step 1 (trained on un-annotated data), and only worry about fine-tuning it for step 2. [\nSource\nfor book icon].\nBERT builds on top of a number of clever ideas that have been bubbling up in the NLP community recently – including but not limited to\nSemi-supervised Sequence Learning\n(by\nAndrew Dai\nand\nQuoc Le\n)\n,\nELMo\n(by\nMatthew Peters\nand researchers from\nAI2\nand\nUW CSE\n)\n,\nULMFiT\n(by fast.ai founder\nJeremy Howard\nand\nSebastian Ruder\n)\n, the\nOpenAI transformer\n(by OpenAI researchers\nRadford\n,\nNarasimhan\n,\nSalimans\n, and\nSutskever\n)\n, and the Transformer\n(\nVaswani et al\n)\n.\nThere are a number of concepts one needs to be aware of to properly wrap one’s head around what BERT is. So let’s start by looking at ways you can use BERT before looking at the concepts involved in the model itself.\nExample: Sentence Classification\nThe most straight-forward way to use BERT is to use it to classify a single piece of text. This model would look like this:\nTo train such a model, you mainly have to train the classifier, with minimal changes happening to the BERT model during the training phase. This training process is called Fine-Tuning, and has roots in\nSemi-supervised Sequence Learning\nand ULMFiT.\nFor people not versed in the topic, since we’re talking about classifiers, then we are in the supervised-learning domain of machine learning. Which would mean we need a labeled dataset to train such a model. For this spam classifier example, the labeled dataset would be a list of email messages and a label (“spam” or “not spam” for each message).\nOther examples for such a use-case include:\nSentiment analysis\nInput: Movie/Product review. Output: is the review positive or negative?\nExample dataset:\nSST\nFact-checking\nInput: sentence. Output: “Claim” or “Not Claim”\nMore ambitious/futuristic example:\nInput: Claim sentence. Output: “True” or “False”\nFull Fact\nis an organization building automatic fact-checking tools for the benefit of the public. Part of their pipeline is a classifier that reads news articles and detects claims (classifies text as either “claim” or “not claim”) which can later be fact-checked (by humans now, with ML later, hopefully).\nVideo:\nSentence embeddings for automated factchecking - Lev Konstantinovskiy\n.\nModel Architecture\nNow that you have an example use-case in your head for how BERT can be used, let’s take a closer look at how it works.\nThe paper presents two model sizes for BERT:\nBERT BASE – Comparable in size to the OpenAI Transformer in order to compare performance\nBERT LARGE – A ridiculously huge model which achieved the state of the art results reported in the paper\nBERT is basically a trained Transformer Encoder stack. This is a good time to direct you to read my earlier post\nThe Illustrated Transformer\nwhich explains the Transformer model – a foundational concept for BERT and the concepts we’ll discuss next.\nBoth BERT model sizes have a large number of encoder layers (which the paper calls Transformer Blocks) – twelve for the Base version, and twenty four for the Large version. These also have larger feedforward-networks (768 and 1024 hidden units respectively), and more attention heads (12 and 16 respectively) than the default configuration in the reference implementation of the Transformer in the initial paper (6 encoder layers, 512 hidden units, and 8 attention heads).\nModel Inputs\nThe first input token is supplied with a special [CLS] token for reasons that will become apparent later on. CLS here stands for Classification.\nJust like the vanilla encoder of the transformer, BERT takes a sequence of words as input which keep flowing up the stack. Each layer applies self-attention, and passes its results through a feed-forward network, and then hands it off to the next encoder.\nIn terms of architecture, this has been identical to the Transformer up until this point (aside from size, which are just configurations we can set). It is at the output that we first start seeing how things diverge.\nModel Outputs\nEach position outputs a vector of size\nhidden_size\n(768 in BERT Base). For the sentence classification example we’ve looked at above, we focus on the output of only the first position (that we passed the special [CLS] token to).\nThat vector can now be used as the input for a classifier of our choosing. The paper achieves great results by just using a single-layer neural network as the classifier.\nIf you have more labels (for example if you’re an email service that tags emails with “spam”, “not spam”, “social”, and “promotion”), you just tweak the classifier network to have more output neurons that then pass through softmax.\nParallels with Convolutional Nets\nFor those with a background in computer vision, this vector hand-off should be reminiscent of what happens between the convolution part of a network like VGGNet and the fully-connected classification portion at the end of the network.\nA New Age of Embedding\nThese new developments carry with them a new shift in how words are encoded. Up until now, word-embeddings have been a major force in how leading NLP models deal with language. Methods like Word2Vec and Glove have been widely used for such tasks. Let’s recap how those are used before pointing to what has now changed.\nWord Embedding Recap\nFor words to be processed by machine learning models, they need some form of numeric representation that models can use in their calculation. Word2Vec showed that we can use a vector (a list of numbers) to properly represent words in a way that captures\nsemantic\nor meaning-related relationships (e.g. the ability to tell if words are similar, or opposites, or that a pair of words like “Stockholm” and “Sweden” have the same relationship between them as “Cairo” and “Egypt” have between them) as well as syntactic, or grammar-based, relationships (e.g. the relationship between “had” and “has” is the same as that between “was” and “is”).\nThe field quickly realized it’s a great idea to use embeddings that were pre-trained on vast amounts of text data instead of training them alongside the model on what was frequently a small dataset. So it became possible to download a list of words and their embeddings generated by pre-training with Word2Vec or GloVe. This is an example of the GloVe embedding of the word “stick” (with an embedding vector size of 200)\nThe GloVe word embedding of the word \"stick\" - a vector of 200 floats (rounded to two decimals). It goes on for two hundred values.\nSince these are large and full of numbers, I use the following basic shape in the figures in my posts to show vectors:\nELMo: Context Matters\nIf we’re using this GloVe representation, then the word “stick” would be represented by this vector no-matter what the context was. “Wait a minute” said a number of NLP researchers\n(\nPeters et. al., 2017\n,\nMcCann et. al., 2017\n, and yet again\nPeters et. al., 2018 in the ELMo paper\n)\n, “\nstick\n”” has multiple meanings depending on where it’s used. Why not give it an embedding based on the context it’s used in – to both capture the word meaning in that context as well as other contextual information?”. And so,\ncontextualized\nword-embeddings were born.\nContextualized word-embeddings can give words different embeddings based on the meaning they carry in the context of the sentence. Also,\nRIP Robin Williams\nInstead of using a fixed embedding for each word, ELMo looks at the entire sentence before assigning each word in it an embedding. It uses a bi-directional LSTM trained on a specific task to be able to create those embeddings.\nELMo provided a significant step towards pre-training in the context of NLP. The ELMo LSTM would be trained on a massive dataset in the language of our dataset, and then we can use it as a component in other models that need to handle language.\nWhat’s ELMo’s secret?\nELMo gained its language understanding from being trained to predict the next word in a sequence of words - a task called\nLanguage Modeling\n. This is convenient because we have vast amounts of text data that such a model can learn from without needing labels.\nA step in the pre-training process of ELMo: Given “Let’s stick to” as input, predict the next most likely word – a\nlanguage modeling\ntask. When trained on a large dataset, the model starts to pick up on language patterns. It’s unlikely it’ll accurately guess the next word in this example. More realistically, after a word such as “hang”, it will assign a higher probability to a word like “out” (to spell “hang out”) than to “camera”.\nWe can see the hidden state of each unrolled-LSTM step peaking out from behind ELMo’s head. Those come in handy in the embedding process after this pre-training is done.\nELMo actually goes a step further and trains a bi-directional LSTM – so that its language model doesn’t only have a sense of the next word, but also the previous word.\nGreat slides\non ELMo\nELMo comes up with the contextualized embedding through grouping together the hidden states (and initial embedding) in a certain way (concatenation followed by weighted summation).\nULM-FiT: Nailing down Transfer Learning in NLP\nULM-FiT introduced methods to effectively utilize a lot of what the model learns during pre-training – more than just embeddings, and more than contextualized embeddings. ULM-FiT introduced a language model and a process to effectively fine-tune that language model for various tasks.\nNLP finally had a way to do transfer learning probably as well as Computer Vision could.\nThe Transformer: Going beyond LSTMs\nThe release of the Transformer paper and code, and the results it achieved on tasks such as machine translation started to make some in the field think of them as a replacement to LSTMs. This was compounded by the fact that Transformers deal with long-term dependancies better than LSTMs.\nThe Encoder-Decoder structure of the transformer made it perfect for machine translation. But how would you use it for sentence classification? How would you use it to pre-train a language model that can be fine-tuned for other tasks (\ndownstream\ntasks is what the field calls those supervised-learning tasks that utilize a pre-trained model or component).\nOpenAI Transformer: Pre-training a Transformer Decoder for Language Modeling\nIt turns out we don’t need an entire Transformer to adopt transfer learning and a fine-tunable language model for NLP tasks. We can do with just the decoder of the transformer. The decoder is a good choice because it’s a natural choice for language modeling (predicting the next word) since it’s built to mask future tokens – a valuable feature when it’s generating a translation word by word.\nThe OpenAI Transformer is made up of the decoder stack from the Transformer\nThe model stacked twelve decoder layers. Since there is no encoder in this set up, these decoder layers would not have the encoder-decoder attention sublayer that vanilla transformer decoder layers have. It would still have the self-attention layer, however (masked so it doesn’t peak at future tokens).\nWith this structure, we can proceed to train the model on the same language modeling task: predict the next word using massive (unlabeled) datasets. Just, throw the text of 7,000 books at it and have it learn! Books are great for this sort of task since it allows the model to learn to associate related information even if they’re separated by a lot of text – something you don’t get for example, when you’re training with tweets, or articles.\nThe OpenAI Transformer is now ready to be trained to predict the next word on a dataset made up of 7,000 books.\nTransfer Learning to Downstream Tasks\nNow that the OpenAI transformer is pre-trained and its layers have been tuned to reasonably handle language, we can start using it for downstream tasks. Let’s first look at sentence classification (classify an email message as “spam” or “not spam”):\nHow to use a pre-trained OpenAI transformer to do sentence clasification\nThe OpenAI paper outlines a number of input transformations to handle the inputs for different types of tasks. The following image from the paper shows the structures of the models and input transformations to carry out different tasks.\nIsn’t that clever?\nBERT: From Decoders to Encoders\nThe openAI transformer gave us a fine-tunable pre-trained model based on the Transformer. But something went missing in this transition from LSTMs to Transformers. ELMo’s language model was bi-directional, but the openAI transformer only trains a forward language model. Could we build a transformer-based model whose language model looks both forward and backwards (in the technical jargon – “is conditioned on both left and right context”)?\n“Hold my beer”, said R-rated BERT.\nMasked Language Model\n“We’ll use transformer encoders”, said BERT.\n“This is madness”, replied Ernie, “Everybody knows bidirectional conditioning would allow each word to indirectly see itself in a multi-layered context.”\n“We’ll use masks”, said BERT confidently.\nBERT's clever language modeling task masks 15% of words in the input and asks the model to predict the missing word.\nFinding the right task to train a Transformer stack of encoders is a complex hurdle that BERT resolves by adopting a “masked language model” concept from earlier literature (where it’s called a Cloze task).\nBeyond masking 15% of the input, BERT also mixes things a bit in order to improve how the model later fine-tunes. Sometimes it randomly replaces a word with another word and asks the model to predict the correct word in that position.\nTwo-sentence Tasks\nIf you look back up at the input transformations the OpenAI transformer does to handle different tasks, you’ll notice that some tasks require the model to say something intelligent about two sentences (e.g. are they simply paraphrased versions of each other? Given a wikipedia entry as input, and a question regarding that entry as another input, can we answer that question?).\nTo make BERT better at handling relationships between multiple sentences, the pre-training process includes an additional task: Given two sentences (A and B), is B likely to be the sentence that follows A, or not?\nThe second task BERT is pre-trained on is a two-sentence classification task. The tokenization is oversimplified in this graphic as BERT actually uses WordPieces as tokens rather than words --- so some words are broken down into smaller chunks.\nTask specific-Models\nThe BERT paper shows a number of ways to use BERT for different tasks.\nBERT for feature extraction\nThe fine-tuning approach isn’t the only way to use BERT. Just like ELMo, you can use the pre-trained BERT to create contextualized word embeddings. Then you can feed these embeddings to your existing model – a process the paper shows yield results not far behind fine-tuning BERT on a task such as named-entity recognition.\nWhich vector works best as a contextualized embedding? I would think it depends on the task. The paper examines six choices (Compared to the fine-tuned model which achieved a score of 96.4):\nTake BERT out for a spin\nThe best way to try out BERT is through the\nBERT FineTuning with Cloud TPUs\nnotebook hosted on Google Colab. If you’ve never used Cloud TPUs before, this is also a good starting point to try them as well as the BERT code works on TPUs, CPUs and GPUs as well.\nThe next step would be to look at the code in the\nBERT repo\n:\nThe model is constructed in\nmodeling.py\n(\nclass BertModel\n) and is pretty much identical to a vanilla Transformer encoder.\nrun_classifier.py\nis an example of the fine-tuning process. It also constructs the classification layer for the supervised model. If you want to construct your own classifier, check out the\ncreate_model()\nmethod in that file.\nSeveral pre-trained models are available for download. These span BERT Base and BERT Large, as well as languages such as English, Chinese, and a multi-lingual model covering 102 languages trained on wikipedia.\nBERT doesn’t look at words as tokens. Rather, it looks at WordPieces.\ntokenization.py\nis the tokenizer that would turns your words into wordPieces appropriate for BERT.\nYou can also check out the\nPyTorch implementation of BERT\n. The\nAllenNLP\nlibrary uses this implementation to\nallow using BERT embeddings\nwith any model.\nAcknowledgements\nThanks to\nJacob Devlin\n,\nMatt Gardner\n,\nKenton Lee\n,\nMark Neumann\n, and\nMatthew Peters\nfor providing feedback on earlier drafts of this post.\nWritten on December 3, 2018", "metadata": { "title": "The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.", "description": "Discussions:\nHacker News (98 points, 19 comments), Reddit r/MachineLearning (164 points, 20 comments)\n\n\nTranslations: Chinese (Simplified), French 1, French 2, Japanese, Korean, Persian, Russian, Spanish\n\n2021 Update: I created this brief and highly accessible video intro to BERT\n\n\n\n\n\nThe year 2018 has been an inflection point for machine learning models handling text (or more accurately, Natural Language Processing or NLP for short). Our conceptual understanding of how best to represent words and sentences in a way that best captures underlying meanings and relationships is rapidly evolving. Moreover, the NLP community has been putting forward incredibly powerful components that you can freely download and use in your own models and pipelines (It’s been referred to as NLP’s ImageNet moment, referencing how years ago similar developments accelerated the development of machine learning in Computer Vision tasks).", "keywords": null, "author": "Jay Alammar", "og:description": "Discussions:\nHacker News (98 points, 19 comments), Reddit r/MachineLearning (164 points, 20 comments)\n\n\nTranslations: Chinese (Simplified), French 1, French 2, Japanese, Korean, Persian, Russian, Spanish\n\n2021 Update: I created this brief and highly accessible video intro to BERT\n\n\n\n\n\nThe year 2018 has been an inflection point for machine learning models handling text (or more accurately, Natural Language Processing or NLP for short). Our conceptual understanding of how best to represent words and sentences in a way that best captures underlying meanings and relationships is rapidly evolving. Moreover, the NLP community has been putting forward incredibly powerful components that you can freely download and use in your own models and pipelines (It’s been referred to as NLP’s ImageNet moment, referencing how years ago similar developments accelerated the development of machine learning in Computer Vision tasks).", "og:title": "The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning)" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://jalammar.github.io/images/bert-transfer-learning.png", "bytes": 256404, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-7d11dc70e9bb1b255140.png", "src": "https://jalammar.github.io/images/bert-transfer-learning.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/BERT-classification-spam.png", "bytes": 84351, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-1b85d104587ac30e342c.png", "src": "https://jalammar.github.io/images/BERT-classification-spam.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/bert-base-bert-large.png", "bytes": 64358, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-72cd37d361b052096c83.png", "src": "https://jalammar.github.io/images/bert-base-bert-large.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/bert-base-bert-large-encoders.png", "bytes": 76925, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-3a1a53e6263e3e93f7e9.png", "src": "https://jalammar.github.io/images/bert-base-bert-large-encoders.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/bert-input-output.png", "bytes": 51503, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-45c6746366afdadd4e77.png", "src": "https://jalammar.github.io/images/bert-input-output.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/bert-encoders-input.png", "bytes": 57236, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-048894d81d5eebe9579d.png", "src": "https://jalammar.github.io/images/bert-encoders-input.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/bert-output-vector.png", "bytes": 59920, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-5eb8f7109853ed1892cf.png", "src": "https://jalammar.github.io/images/bert-output-vector.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/bert-classifier.png", "bytes": 80313, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-f75eb4705bdc992a2adf.png", "src": "https://jalammar.github.io/images/bert-classifier.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/vgg-net-classifier.png", "bytes": 431500, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-cb004af56096265aeee6.png", "src": "https://jalammar.github.io/images/vgg-net-classifier.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/glove-embedding.png", "bytes": 14488, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-66c0f2327d4cdeb91d34.png", "src": "https://jalammar.github.io/images/glove-embedding.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/vector-boxes.png", "bytes": 6911, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-5f5f9c57dd771c98df4a.png", "src": "https://jalammar.github.io/images/vector-boxes.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/elmo-embedding-robin-williams.png", "bytes": 873103, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-355fa1376dfab587a432.png", "src": "https://jalammar.github.io/images/elmo-embedding-robin-williams.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/elmo-word-embedding.png", "bytes": 63419, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-9cdd7481fa7b167dfee6.png", "src": "https://jalammar.github.io/images/elmo-word-embedding.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/Bert-language-modeling.png", "bytes": 98024, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-1f10d34a1ad7e655bdce.png", "src": "https://jalammar.github.io/images/Bert-language-modeling.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/elmo-forward-backward-language-model-embedding.png", "bytes": 134770, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-00156fe104700074c258.png", "src": "https://jalammar.github.io/images/elmo-forward-backward-language-model-embedding.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/elmo-embedding.png", "bytes": 158617, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-a2a0df35f30d313a51d9.png", "src": "https://jalammar.github.io/images/elmo-embedding.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/openai-transformer-1.png", "bytes": 80876, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-1dfc3c6e7c3e8cc45709.png", "src": "https://jalammar.github.io/images/openai-transformer-1.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/openai-transformer-language-modeling.png", "bytes": 74391, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-bb1f6ed744803071458b.png", "src": "https://jalammar.github.io/images/openai-transformer-language-modeling.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/openai-transformer-sentence-classification.png", "bytes": 102985, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-9a2afed3243ce15d9294.png", "src": "https://jalammar.github.io/images/openai-transformer-sentence-classification.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/openai-input transformations.png", "bytes": 160508, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-00c91c1088b2e1401fef.png", "src": "https://jalammar.github.io/images/openai-input transformations.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/BERT-language-modeling-masked-lm.png", "bytes": 121914, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-95af5faf02ed5d706c61.png", "src": "https://jalammar.github.io/images/BERT-language-modeling-masked-lm.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/bert-next-sentence-prediction.png", "bytes": 107183, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-5b823ce9129977e1659c.png", "src": "https://jalammar.github.io/images/bert-next-sentence-prediction.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/bert-tasks.png", "bytes": 153716, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-5653611462e31b9e6ce2.png", "src": "https://jalammar.github.io/images/bert-tasks.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/bert-contexualized-embeddings.png", "bytes": 117189, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-83c0e55a83e3cd270274.png", "src": "https://jalammar.github.io/images/bert-contexualized-embeddings.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://jalammar.github.io/images/bert-feature-extraction-contextualized-embeddings.png", "bytes": 101554, "reused": false, "local_path": "attachments/84b0f50c3cc1/images/body-768dc4bd8263f22bc0d7.png", "src": "https://jalammar.github.io/images/bert-feature-extraction-contextualized-embeddings.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=ioGry-89gqE", "src": "https://www.youtube.com/embed/ioGry-89gqE", "status": "ok", "kind": "single_video", "local_path": "attachments/84b0f50c3cc1/videos/repair-ioGry-89gqE.mp4", "bytes": 20129250, "valid": true, "height": 1080, "width": 1920, "duration": "369.881000", "reused": false }, { "url": "https://www.youtube.com/watch?v=ddf0lgPCoSo", "src": "https://www.youtube.com/watch?v=ddf0lgPCoSo", "status": "ok", "kind": "single_video", "local_path": "attachments/84b0f50c3cc1/videos/repair-ddf0lgPCoSo.mp4", "bytes": 124117694, "valid": true, "height": 1080, "width": 1920, "duration": "2260.681000", "reused": false }, { "url": "https://www.youtube.com/watch?v=OwwdgsN9wF8", "src": "https://www.youtube.com/watch?v=OwwdgsN9wF8", "status": "source_unavailable", "kind": "single_video", "error": "ERROR: [youtube] OwwdgsN9wF8: Video unavailable\n", "title": "", "retried_at": "2026-10-20T18:07:11Z" } ], "links": [ { "href": "https://news.ycombinator.com/item?id=18751469", "label": "Hacker News (98 points, 19 comments)", "type": "a" }, { "href": "https://www.reddit.com/r/MachineLearning/comments/a3ykzf/r_the_illustrated_bert_and_elmo_how_nlp_cracked/", "label": "Reddit r/MachineLearning (164 points, 20 comments)", "type": "a" }, { "href": "https://blog.csdn.net/qq_41664845/article/details/84787969", "label": "Chinese (Simplified)", "type": "a" }, { "href": "https://a-coles.github.io/2020/11/15/bert-illustre.html", "label": "French 1", "type": "a" }, { "href": "https://lbourdois.github.io/blog/nlp/BERT/", "label": "French 2", "type": "a" }, { "href": "https://tech-magazine.opt.ne.jp/entry/2020/05/01/132654", "label": "Japanese", "type": "a" }, { "href": "https://nlpinkorean.github.io/illustrated-bert/", "label": "Korean", "type": "a" }, { "href": "http://blog.class.vision/1397/09/bert-in-nlp/", "label": "Persian", "type": "a" }, { "href": "https://habr.com/ru/post/487358/", "label": "Russian", "type": "a" }, { "href": "https://www.ibidem-translations.com/edu/traduccion-bert-elmo-pnl/", "label": "Spanish", "type": "a" }, { "href": "https://www.youtube.com/embed/ioGry-89gqE", "label": "", "type": "iframe" }, { "href": "http://ruder.io/nlp-imagenet/", "label": "NLP’s ImageNet moment", "type": "a" }, { "href": "https://ai.googleblog.com/2018/11/open-sourcing-bert-state-of-art-pre.html", "label": "release", "type": "a" }, { "href": "https://github.com/google-research/bert", "label": "BERT", "type": "a" }, { "href": "https://twitter.com/lmthang/status/1050543868041555969", "label": "described", "type": "a" }, { "href": "https://commons.wikimedia.org/wiki/File:Documents_icon_-_noun_project_5020.svg", "label": "Source", "type": "a" }, { "href": "https://arxiv.org/abs/1511.01432", "label": "Semi-supervised Sequence Learning", "type": "a" }, { "href": "https://twitter.com/iamandrewdai", "label": "Andrew Dai", "type": "a" }, { "href": "https://twitter.com/quocleix", "label": "Quoc Le", "type": "a" }, { "href": "https://arxiv.org/abs/1802.05365", "label": "ELMo", "type": "a" }, { "href": "https://twitter.com/mattthemathman", "label": "Matthew Peters", "type": "a" }, { "href": "https://allenai.org/", "label": "AI2", "type": "a" }, { "href": "https://www.engr.washington.edu/about/bldgs/cse", "label": "UW CSE", "type": "a" }, { "href": "https://arxiv.org/abs/1801.06146", "label": "ULMFiT", "type": "a" }, { "href": "https://twitter.com/jeremyphoward", "label": "Jeremy Howard", "type": "a" }, { "href": "https://twitter.com/seb_ruder", "label": "Sebastian Ruder", "type": "a" }, { "href": "https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf", "label": "OpenAI transformer", "type": "a" }, { "href": "https://twitter.com/alecrad", "label": "Radford", "type": "a" }, { "href": "https://twitter.com/karthik_r_n", "label": "Narasimhan", "type": "a" }, { "href": "https://twitter.com/timsalimans", "label": "Salimans", "type": "a" }, { "href": "https://twitter.com/ilyasut", "label": "Sutskever", "type": "a" }, { "href": "https://arxiv.org/pdf/1706.03762.pdf", "label": "Vaswani et al", "type": "a" }, { "href": "https://nlp.stanford.edu/sentiment/", "label": "SST", "type": "a" }, { "href": "https://fullfact.org/", "label": "Full Fact", "type": "a" }, { "href": "https://www.youtube.com/watch?v=ddf0lgPCoSo", "label": "Sentence embeddings for automated factchecking - Lev Konstantinovskiy", "type": "a" }, { "href": "https://jalammar.github.io/illustrated-transformer/", "label": "The Illustrated Transformer", "type": "a" }, { "href": "https://arxiv.org/abs/1705.00108", "label": "Peters et. al., 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1708.00107", "label": "McCann et. al., 2017", "type": "a" }, { "href": "https://arxiv.org/pdf/1802.05365.pdf", "label": "Peters et. al., 2018 in the ELMo paper", "type": "a" }, { "href": "https://www.youtube.com/watch?v=OwwdgsN9wF8", "label": "RIP Robin Williams", "type": "a" }, { "href": "https://www.slideshare.net/shuntaroy/a-review-of-deep-contextualized-word-representations-peters-2018", "label": "Great slides", "type": "a" }, { "href": "https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb", "label": "BERT FineTuning with Cloud TPUs", "type": "a" }, { "href": "https://github.com/google-research/bert/blob/master/modeling.py", "label": "modeling.py", "type": "a" }, { "href": "https://github.com/google-research/bert/blob/master/run_classifier.py", "label": "run_classifier.py", "type": "a" }, { "href": "https://github.com/google-research/bert/blob/master/tokenization.py", "label": "tokenization.py", "type": "a" }, { "href": "https://github.com/huggingface/pytorch-pretrained-BERT", "label": "PyTorch implementation of BERT", "type": "a" }, { "href": "https://github.com/allenai/allennlp", "label": "AllenNLP", "type": "a" }, { "href": "https://github.com/allenai/allennlp/pull/2067", "label": "allow using BERT embeddings", "type": "a" }, { "href": "https://github.com/jacobdevlin-google", "label": "Jacob Devlin", "type": "a" }, { "href": "https://twitter.com/nlpmattg", "label": "Matt Gardner", "type": "a" }, { "href": "https://github.com/kentonl", "label": "Kenton Lee", "type": "a" }, { "href": "https://twitter.com/markneumannnn", "label": "Mark Neumann", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:26Z", "source_url": "https://jalammar.github.io/illustrated-bert/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/1511.01432", "bytes": 98163, "reused": false, "local_path": "papers/repair/cf073381efc2b3d5b9c5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1802.05365", "bytes": 425880, "reused": true, "local_path": "papers/repair/c4dc3d740cad43df1dd2.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1801.06146", "bytes": 979417, "reused": true, "local_path": "papers/repair/f99f602eebf2965d0ed0.pdf" }, { "status": "http_403", "url": "https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf", "retried_at": "2026-10-20T18:07:10Z" }, { "status": "ok", "url": "https://arxiv.org/pdf/1706.03762.pdf", "bytes": 2215244, "reused": false, "local_path": "papers/repair/10f9519a4068b56b1057.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1705.00108", "bytes": 401085, "reused": false, "local_path": "papers/repair/c992678426b7170fbce5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1708.00107", "bytes": 446263, "reused": false, "local_path": "papers/repair/e2c0a99965fb6abdde60.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1802.05365.pdf", "bytes": 425880, "reused": false, "local_path": "papers/repair/bb088fd0138ae4825b12.pdf" } ], "actual_video_files": [], "body_chars": 18311, "expected_body_images": 25, "discovery_status": "ok", "delivery_status": "partial", "missing_reason": "http_403；source_unavailable", "raw_html": "attachments/84b0f50c3cc1/pages/source.html", "local_html": "attachments/84b0f50c3cc1/pages/article.html", "body_base_path": "attachments/84b0f50c3cc1/pages/", "body_markdown_path": "attachments/84b0f50c3cc1/pages/article.md" }