{ "id": "607baf97b999", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/", "source_excel_row": 1602, "source_url": "https://lilianweng.github.io/posts/2019-01-31-lm/", "is_paid_flag": "", "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/elmo-and-bert.png", "bytes": 929637, "reused": true, "local_path": "attachments/607baf97b999/images/body-5e717a51b0a5329fa972.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/elmo-and-bert.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/nmt-recap.png", "bytes": 268193, "reused": true, "local_path": "attachments/607baf97b999/images/body-bcfb5637b30440671005.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/nmt-recap.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/CoVe.png", "bytes": 183059, "reused": true, "local_path": "attachments/607baf97b999/images/body-7365015a0a10cc69acd8.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/CoVe.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ELMo-biLSTM.png", "bytes": 473871, "reused": true, "local_path": "attachments/607baf97b999/images/body-9d994d94aa3f953655fe.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/ELMo-biLSTM.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/CVT.png", "bytes": 244780, "reused": true, "local_path": "attachments/607baf97b999/images/body-4b7c5a8a25f1aef4e33a.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/CVT.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/CVT-example.png", "bytes": 269806, "reused": true, "local_path": "attachments/607baf97b999/images/body-7ad146452781a398f651.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/CVT-example.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ULMFiT.png", "bytes": 1588495, "reused": true, "local_path": "attachments/607baf97b999/images/body-2208e66f87d45609e14a.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/ULMFiT.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/OpenAI-GPT-transformer-decoder.png", "bytes": 516917, "reused": true, "local_path": "attachments/607baf97b999/images/body-e495e282eb1d3f3e9144.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/OpenAI-GPT-transformer-decoder.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT-classification.png", "bytes": 51037, "reused": true, "local_path": "attachments/607baf97b999/images/body-8a0f9752f056eb917bbf.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT-classification.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT-downstream-tasks.png", "bytes": 272909, "reused": true, "local_path": "attachments/607baf97b999/images/body-b96d2353529c40b14aef.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT-downstream-tasks.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/transformer-encoder-2.png", "bytes": 100122, "reused": true, "local_path": "attachments/607baf97b999/images/body-e2ee524fbebc06b6e190.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/transformer-encoder-2.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/language-model-comparison.png", "bytes": 227696, "reused": true, "local_path": "attachments/607baf97b999/images/body-5d14984c36bf66c73897.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/language-model-comparison.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BERT-input-embedding.png", "bytes": 87898, "reused": true, "local_path": "attachments/607baf97b999/images/body-08c8fd1e0c80108aa470.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/BERT-input-embedding.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BERT-downstream-tasks.png", "bytes": 415583, "reused": true, "local_path": "attachments/607baf97b999/images/body-c58896fcdd52fc06c2a0.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/BERT-downstream-tasks.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/T5.png", "bytes": 253294, "reused": true, "local_path": "attachments/607baf97b999/images/body-7970b3c999effb6f02b6.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/T5.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT3-train-data.png", "bytes": 88113, "reused": true, "local_path": "attachments/607baf97b999/images/body-85fc2d78dbd4ba853f15.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT3-train-data.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT3-eval.png", "bytes": 282592, "reused": true, "local_path": "attachments/607baf97b999/images/body-dd77dfe447995839038a.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT3-eval.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/XLNet-two-stream-attention.png", "bytes": 384564, "reused": true, "local_path": "attachments/607baf97b999/images/body-3be7c9763f11993ba7a6.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/XLNet-two-stream-attention.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/XLNet-glue.png", "bytes": 76634, "reused": true, "local_path": "attachments/607baf97b999/images/body-e529afcc9a2bd0fb2fae.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/XLNet-glue.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BART.png", "bytes": 180008, "reused": true, "local_path": "attachments/607baf97b999/images/body-8045a59ac9bcdcc697fc.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/BART.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BART-perf.png", "bytes": 258219, "reused": true, "local_path": "attachments/607baf97b999/images/body-c0273e122159d4c19386.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/BART-perf.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ELECTRA-overview.png", "bytes": 88004, "reused": true, "local_path": "attachments/607baf97b999/images/body-8bf9b1628b286eb863de.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/ELECTRA-overview.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ELECTRA-perf.png", "bytes": 138525, "reused": true, "local_path": "attachments/607baf97b999/images/body-76b71f8459361bdc98c5.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/ELECTRA-perf.png", "alt": "", "downloaded": true } ], "videos": [ { "url": "https://www.youtube.com/watch?v=l5einDQ-Ttc", "src": "https://www.youtube.com/watch?v=l5einDQ-Ttc", "status": "ok", "kind": "single_video", "local_path": "attachments/607baf97b999/videos/repair-l5einDQ-Ttc.mp4", "bytes": 19398754, "valid": true, "height": 1080, "width": 1920, "duration": "139.221000", "reused": false } ], "body_markdown": "[Updated on 2019-02-14: add\nULMFiT\nand\nGPT-2\n.]\n[Updated on 2020-02-29: add\nALBERT\n.]\n[Updated on 2020-10-25: add\nRoBERTa\n.]\n[Updated on 2020-12-13: add\nT5\n.]\n[Updated on 2020-12-30: add\nGPT-3\n.]\n[Updated on 2021-11-13: add\nXLNet\n,\nBART\nand\nELECTRA\n; Also updated the\nSummary\nsection.]\nI guess they are Elmo & Bert? (Image source:\nhere\n)\nWe have seen amazing progress in NLP in 2018. Large-scale pre-trained language modes like\nOpenAI GPT\nand\nBERT\nhave achieved great performance on a variety of language tasks using generic model architectures. The idea is similar to how ImageNet classification pre-training helps many vision tasks (*). Even better than vision classification pre-training, this simple and powerful approach in NLP does not require labeled data for pre-training, allowing us to experiment with increased training scale, up to our very limit.\n(*) He et al. (2018)\nfound\nthat pre-training might not be necessary for image segmentation task.\nIn my previous NLP\npost on word embedding\n, the introduced embeddings are not context-specific — they are learned based on word concurrency but not sequential context. So in two sentences, “\nI am eating an apple\n” and “\nI have an Apple phone\n”, two “apple” words refer to very different things but they would still share the same word embedding vector.\nDespite this, early adoption of word embeddings in problem-solving is to use them as additional features for an existing task-specific model and in a way the improvement is bounded.\nIn this post, we will discuss how various approaches were proposed to make embeddings dependent on context, and to make them easier and cheaper to be applied to downstream tasks in general form.\nCoVe\n#\nCoVe\n(\nMcCann et al. 2017\n), short for\nContextual Word Vectors\n, is a type of word embeddings learned by an encoder in an\nattentional seq-to-seq\nmachine translation model.\nDifferent from traditional word embeddings introduced\nhere\n, CoVe word representations are functions of the entire input sentence.\nNMT Recap\n#\nHere the Neural Machine Translation (\nNMT\n) model is composed of a standard, two-layer, bidirectional LSTM encoder and an attentional two-layer unidirectional LSTM decoder. It is pre-trained on the English-German translation task. The encoder learns and optimizes the embedding vectors of English words in order to translate them to German. With the intuition that the encoder should capture high-level semantic and syntactic meanings before transforming words into another language, the encoder output is used to provide contextualized word embeddings for various downstream language tasks.\nThe NMT base model used in CoVe.\nA sequence of $n$ words in source language (English): $x = [x_1, \\dots, x_n]$.\nA sequence of $m$ words in target language (German): $y = [y_1, \\dots, y_m]$.\nThe\nGloVe\nvectors of source words: $\\text{GloVe}(x)$.\nRandomly initialized embedding vectors of target words: $z = [z_1, \\dots, z_m]$.\nThe biLSTM encoder outputs a sequence of hidden states: $h = [h_1, \\dots, h_n] = \\text{biLSTM}(\\text{GloVe}(x))$ and $h_t = [\\overrightarrow{h}_t; \\overleftarrow{h}_t]$ where the forward LSTM computes $\\overrightarrow{h}_t = \\text{LSTM}(x_t, \\overrightarrow{h}_{t-1})$ and the backward computation gives us $\\overleftarrow{h}_t = \\text{LSTM}(x_t, \\overleftarrow{h}_{t-1})$.\nThe attentional decoder outputs a distribution over words: $p(y_t \\mid H, y_1, \\dots, y_{t-1})$ where $H$ is a stack of hidden states $\\{h\\}$ along the time dimension:\n$$\n\\begin{aligned}\n\\text{decoder hidden state: } s_t &= \\text{LSTM}([z_{t-1}; \\tilde{h}_{t-1}], s_{t-1}) \\\\\n\\text{attention weights: } \\alpha_t &= \\text{softmax}(H(W_1 s_t + b_1)) \\\\\n\\text{context-adjusted hidden state: } \\tilde{h}_t &= \\tanh(W_2[H^\\top\\alpha_t;s_t] + b_2) \\\\\n\\text{decoder output: } p(y_t\\mid H, y_1, \\dots, y_{t-1}) &= \\text{softmax}(W_\\text{out} \\tilde{h}_t + b_\\text{out})\n\\end{aligned}\n$$\nUse CoVe in Downstream Tasks\n#\nThe hidden states of NMT encoder are defined as\ncontext vectors\nfor other language tasks:\n$$\n\\text{CoVe}(x) = \\text{biLSTM}(\\text{GloVe}(x))\n$$\nThe paper proposed to use the concatenation of GloVe and CoVe for question-answering and classification tasks. GloVe learns from the ratios of global word co-occurrences, so it has no sentence context, while CoVe is generated by processing text sequences is able to capture the contextual information.\n$$\nv = [\\text{GloVe}(x); \\text{CoVe}(x)]\n$$\nGiven a downstream task, we first generate the concatenation of GloVe + CoVe vectors of input words and then feed them into the task-specific models as additional features.\nThe CoVe embeddings are generated by an encoder trained for machine translation task. The encoder can be plugged into any downstream task-specific model. (Image source:\noriginal paper\n)\nSummary\n: The limitation of CoVe is obvious: (1) pre-training is bounded by available datasets on the supervised translation task; (2) the contribution of CoVe to the final performance is constrained by the task-specific model architecture.\nIn the following sections, we will see that ELMo overcomes issue (1) by unsupervised pre-training and OpenAI GPT & BERT further overcome both problems by unsupervised pre-training + using generative model architecture for different downstream tasks.\nELMo\n#\nELMo\n, short for\nEmbeddings from Language Model\n(\nPeters, et al, 2018\n) learns contextualized word representation by pre-training a language model in an\nunsupervised\nway.\nBidirectional Language Model\n#\nThe bidirectional Language Model (\nbiLM\n) is the foundation for ELMo. While the input is a sequence of $n$ tokens, $(x_1, \\dots, x_n)$, the language model learns to predict the probability of next token given the history.\nIn the forward pass, the history contains words before the target token,\n$$\np(x_1, \\dots, x_n) = \\prod_{i=1}^n p(x_i \\mid x_1, \\dots, x_{i-1})\n$$\nIn the backward pass, the history contains words after the target token,\n$$\np(x_1, \\dots, x_n) = \\prod_{i=1}^n p(x_i \\mid x_{i+1}, \\dots, x_n)\n$$\nThe predictions in both directions are modeled by multi-layer LSTMs with hidden states $\\overrightarrow{\\mathbf{h}}_{i,\\ell}$ and $\\overleftarrow{\\mathbf{h}}_{i,\\ell}$ for input token $x_i$ at the layer level $\\ell=1,\\dots,L$.\nThe final layer’s hidden state $\\mathbf{h}_{i,L} = [\\overrightarrow{\\mathbf{h}}_{i,L}; \\overleftarrow{\\mathbf{h}}_{i,L}]$ is used to output the probabilities over tokens after softmax normalization. They share the embedding layer and the softmax layer, parameterized by $\\Theta_e$ and $\\Theta_s$ respectively.\nThe biLSTM base model of ELMo. (Image source: recreated based on the figure in [\"Neural Networks, Types, and Functional Programming\"](http://colah.github.io/posts/2015-09-NN-Types-FP/) by Christopher Olah.)\nThe model is trained to minimize the negative log likelihood (= maximize the log likelihood for true words) in both directions:\n$$\n\\begin{aligned}\n\\mathcal{L} = - \\sum_{i=1}^n \\Big( \n\\log p(x_i \\mid x_1, \\dots, x_{i-1}; \\Theta_e, \\overrightarrow{\\Theta}_\\text{LSTM}, \\Theta_s) + \\\\\n\\log p(x_i \\mid x_{i+1}, \\dots, x_n; \\Theta_e, \\overleftarrow{\\Theta}_\\text{LSTM}, \\Theta_s) \\Big)\n\\end{aligned}\n$$\nELMo Representations\n#\nOn top of a $L$-layer biLM, ELMo stacks all the hidden states across layers together by learning a task-specific linear combination. The hidden state representation for the token $x_i$ contains $2L+1$ vectors:\n$$\nR_i = \\{ \\mathbf{h}_{i,\\ell} \\mid \\ell = 0, \\dots, L \\}\n$$\nwhere $\\mathbf{h}_{0, \\ell}$ is the embedding layer output and $\\mathbf{h}_{i, \\ell} = [\\overrightarrow{\\mathbf{h}}_{i,\\ell}; \\overleftarrow{\\mathbf{h}}_{i,\\ell}]$.\nThe weights, $\\mathbf{s}^\\text{task}$, in the linear combination are learned for each end task and normalized by softmax. The scaling factor $\\gamma^\\text{task}$ is used to correct the misalignment between the distribution of biLM hidden states and the distribution of task specific representations.\n$$\nv_i = f(R_i; \\Theta^\\text{task}) = \\gamma^\\text{task} \\sum_{\\ell=0}^L s^\\text{task}_i \\mathbf{h}_{i,\\ell}\n$$\nTo evaluate what kind of information is captured by hidden states across different layers, ELMo is applied on semantic-intensive and syntax-intensive tasks respectively using representations in different layers of biLM:\nSemantic task\n: The\nword sense disambiguation (WSD)\ntask emphasizes the meaning of a word given a context. The biLM top layer is better at this task than the first layer.\nSyntax task\n: The\npart-of-speech\n(POS) tagging\ntask aims to infer the grammatical role of a word in one sentence. A higher accuracy can be achieved by using the biLM first layer than the top layer.\nThe comparison study indicates that syntactic information is better represented at lower layers while semantic information is captured by higher layers. Because different layers tend to carry different type of information,\nstacking them together helps\n.\nUse ELMo in Downstream Tasks\n#\nSimilar to how\nCoVe\ncan help different downstream tasks, ELMo embedding vectors are included in the input or lower levels of task-specific models. Moreover, for some tasks (i.e.,\nSNLI\nand\nSQuAD\n, but not\nSRL\n), adding them into the output level helps too.\nThe improvements brought up by ELMo are largest for tasks with a small supervised dataset. With ELMo, we can also achieve similar performance with much less labeled data.\nSummary\n: The language model pre-training is unsupervised and theoretically the pre-training can be scaled up as much as possible since the unlabeled text corpora are abundant. However, it still has the dependency on task-customized models and thus the improvement is only incremental, while searching for a good model architecture for every task remains non-trivial.\nCross-View Training\n#\nIn ELMo the unsupervised pre-training and task-specific learning happen for two independent models in two separate training stages.\nCross-View Training\n(abbr.\nCVT\n;\nClark et al., 2018\n) combines them into one unified semi-supervised learning procedure where the representation of a biLSTM encoder is improved by both supervised learning with labeled data and unsupervised learning with unlabeled data on auxiliary tasks.\nModel Architecture\n#\nThe model consists of a two-layer bidirectional LSTM encoder and a primary prediction module. During training, the model is fed with labeled and unlabeled data batches alternatively.\nOn\nlabeled examples\n, all the model parameters are updated by standard supervised learning. The loss is the standard cross entropy.\nOn\nunlabeled examples\n, the primary prediction module still can produce a “soft” target, even though we cannot know exactly how accurate they are. In a couple of auxiliary tasks, the predictor only sees and processes a restricted view of the input, such as only using encoder hidden state representation in one direction. The auxiliary task outputs are expected to match the primary prediction target for a full view of input.\nIn this way, the encoder is forced to distill the knowledge of the full context into partial representation. At this stage, the biLSTM encoder is backpropagated but the primary prediction module is\nfixed\n. The loss is to minimize the distance between auxiliary and primary predictions.\nThe overview of semi-supervised language model cross-view training. (Image source:\noriginal paper\n)\nMulti-Task Learning\n#\nWhen training for multiple tasks simultaneously, CVT adds several extra primary prediction models for additional tasks. They all share the same sentence representation encoder.\nDuring supervised training, once one task is randomly selected, parameters in its corresponding predictor and the representation encoder are updated.\nWith unlabeled data samples, the encoder is optimized jointly across all the tasks by minimizing the differences between auxiliary outputs and primary prediction for every task.\nThe multi-task learning encourages better generality of representation and in the meantime produces a nice side-product: all-tasks-labeled examples from unlabeled data. They are precious data labels considering that cross-task labels are useful but fairly rare.\nUse CVT in Downstream Tasks\n#\nTheoretically the primary prediction module can take any form, generic or task-specific design. The examples presented in the CVT paper include both cases.\nIn sequential tagging tasks (classification for every token) like\nNER\nor\nPOS\ntagging, the predictor module contains two fully connected layers and a softmax layer on the output to produce a probability distribution over class labels.\nFor each token $\\mathbf{x}_i$, we take the corresponding hidden states in two layers, $\\mathbf{h}_1^{(i)}$ and $\\mathbf{h}_2^{(i)}$:\n$$\n\\begin{aligned}\np_\\theta(y_i \\mid \\mathbf{x}_i) \n&= \\text{NN}(\\mathbf{h}^{(i)}) \\\\\n&= \\text{NN}([\\mathbf{h}_1^{(i)}; \\mathbf{h}_2^{(i)}]) \\\\\n&= \\text{softmax} \\big( \\mathbf{W}\\cdot\\text{ReLU}(\\mathbf{W'}\\cdot[\\mathbf{h}_1^{(i)}; \\mathbf{h}_2^{(i)}]) + \\mathbf{b} \\big)\n\\end{aligned}\n$$\nThe auxiliary tasks are only fed with forward or backward LSTM state in the first layer. Because they only observe partial context, either on the left or right, they have to learn like a language model, trying to predict the next token given the context. The\nfwd\nand\nbwd\nauxiliary tasks only take one direction. The\nfuture\nand\npast\ntasks take one step further in forward and backward direction, respectively.\n$$\n\\begin{aligned}\np_\\theta^\\text{fwd}(y_i \\mid \\mathbf{x}_i) &= \\text{NN}^\\text{fwd}(\\overrightarrow{\\mathbf{h}}^{(i)}) \\\\\np_\\theta^\\text{bwd}(y_i \\mid \\mathbf{x}_i) &= \\text{NN}^\\text{bwd}(\\overleftarrow{\\mathbf{h}}^{(i)}) \\\\\np_\\theta^\\text{future}(y_i \\mid \\mathbf{x}_i) &= \\text{NN}^\\text{future}(\\overrightarrow{\\mathbf{h}}^{(i-1)}) \\\\\np_\\theta^\\text{past}(y_i \\mid \\mathbf{x}_i) &= \\text{NN}^\\text{past}(\\overleftarrow{\\mathbf{h}}^{(i+1)})\n\\end{aligned}\n$$\nThe sequential tagging task depends on four auxiliary prediction models, their inputs only involving hidden states in one direction: forward, backward, future and past. (Image source:\noriginal paper\n)\nNote that if the primary prediction module has dropout, the dropout layer works as usual when training with labeled data, but it is not applied when generating “soft” target for auxiliary tasks during training with unlabeled data.\nIn the machine translation task, the primary prediction module is replaced with a standard unidirectional LSTM decoder with attention. There are two auxiliary tasks: (1) apply dropout on the attention weight vector by randomly zeroing out some values; (2) predict the future word in the target sequence. The primary prediction for auxiliary tasks to match is the best predicted target sequence produced by running the fixed primary decoder on the input sequence with\nbeam search\n.\nULMFiT\n#\nThe idea of using generative pretrained LM + task-specific fine-tuning was first explored in ULMFiT (\nHoward & Ruder, 2018\n), directly motivated by the success of using ImageNet pre-training for computer vision tasks. The base model is\nAWD-LSTM\n.\nULMFiT follows three steps to achieve good transfer learning results on downstream language classification tasks:\nGeneral LM pre-training\n: on Wikipedia text.\nTarget task LM fine-tuning\n: ULMFiT proposed two training techniques for stabilizing the fine-tuning process. See below.\nDiscriminative fine-tuning\nis motivated by the fact that different layers of LM capture different types of information (see\ndiscussion\nabove). ULMFiT proposed to tune each layer with different learning rates, $\\{\\eta^1, \\dots, \\eta^\\ell, \\dots, \\eta^L\\}$, where $\\eta$ is the base learning rate for the first layer, $\\eta^\\ell$ is for the $\\ell$-th layer and there are $L$ layers in total.\nSlanted triangular learning rates (STLR)\nrefer to a special learning rate scheduling that first linearly increases the learning rate and then linearly decays it. The increase stage is short so that the model can converge to a parameter space suitable for the task fast, while the decay period is long allowing for better fine-tuning.\nTarget task classifier fine-tuning\n: The pretrained LM is augmented with two standard feed-forward layers and a softmax normalization at the end to predict a target label distribution.\nConcat pooling\nextracts max-polling and mean-pooling over the history of hidden states and concatenates them with the final hidden state.\nGradual unfreezing\nhelps to avoid catastrophic forgetting by gradually unfreezing the model layers starting from the last one. First the last layer is unfrozen and fine-tuned for one epoch. Then the next lower layer is unfrozen. This process is repeated until all the layers are tuned.\nThree training stages of ULMFiT. (Image source:\noriginal paper\n)\nGPT\n#\nFollowing the similar idea of ELMo, OpenAI\nGPT\n, short for\nGenerative Pre-training Transformer\n(\nRadford et al., 2018\n), expands the unsupervised language model to a much larger scale by training on a giant collection of free text corpora. Despite of the similarity, GPT has two major differences from ELMo.\nThe model architectures are different: ELMo uses a shallow concatenation of independently trained left-to-right and right-to-left multi-layer LSTMs, while GPT is a multi-layer transformer decoder.\nThe use of contextualized embeddings in downstream tasks are different: ELMo feeds embeddings into models customized for specific tasks as additional features, while GPT fine-tunes the same base model for all end tasks.\nTransformer Decoder as Language Model\n#\nCompared to the\noriginal transformer\narchitecture, the\ntransformer decoder\nmodel discards the encoder part, so there is only one single input sentence rather than two separate source and target sequences.\nThis model applies multiple transformer blocks over the embeddings of input sequences. Each block contains a masked\nmulti-headed self-attention\nlayer and a\npointwise feed-forward\nlayer. The final output produces a distribution over target tokens after softmax normalization.\nThe transformer decoder model architecture in OpenAI GPT.\nThe loss is the negative log-likelihood, same as\nELMo\n, but without backward computation. Let’s say, the context window of the size $k$ is located before the target word and the loss would look like:\n$$\n\\mathcal{L}_\\text{LM} = -\\sum_{i} \\log p(x_i\\mid x_{i-k}, \\dots, x_{i-1})\n$$\nByte Pair Encoding\n#\nByte Pair Encoding\n(\nBPE\n) is used to encode the input sequences. BPE was originally proposed as a data compression algorithm in 1990s and then was adopted to solve the open-vocabulary issue in machine translation, as we can easily run into rare and unknown words when translating into a new language. Motivated by the intuition that rare and unknown words can often be decomposed into multiple subwords, BPE finds the best word segmentation by iteratively and greedily merging frequent pairs of characters.\nSupervised Fine-Tuning\n#\nThe most substantial upgrade that OpenAI GPT proposed is to get rid of the task-specific model and use the pre-trained language model directly!\nLet’s take classification as an example. Say, in the labeled dataset, each input has $n$ tokens, $\\mathbf{x} = (x_1, \\dots, x_n)$, and one label $y$. GPT first processes the input sequence $\\mathbf{x}$ through the pre-trained transformer decoder and the last layer output for the last token $x_n$ is $\\mathbf{h}_L^{(n)}$. Then with only one new trainable weight matrix $\\mathbf{W}_y$, it can predict a distribution over class labels.\n$$\nP(y\\mid x_1, \\dots, x_n) = \\text{softmax}(\\mathbf{h}_L^{(n)}\\mathbf{W}_y)\n$$\nThe loss is to minimize the negative log-likelihood for true labels. In addition, adding the LM loss as an auxiliary loss is found to be beneficial, because:\n(1) it helps accelerate convergence during training and\n(2) it is expected to improve the generalization of the supervised model.\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{cls} &= \\sum_{(\\mathbf{x}, y) \\in \\mathcal{D}} \\log P(y\\mid x_1, \\dots, x_n) = \\sum_{(\\mathbf{x}, y) \\in \\mathcal{D}} \\log \\text{softmax}(\\mathbf{h}_L^{(n)}(\\mathbf{x})\\mathbf{W}_y) \\\\\n\\mathcal{L}_\\text{LM} &= -\\sum_{i} \\log p(x_i\\mid x_{i-k}, \\dots, x_{i-1}) \\\\\n\\mathcal{L} &= \\mathcal{L}_\\text{cls} + \\lambda \\mathcal{L}_\\text{LM}\n\\end{aligned}\n$$\nWith similar designs, no customized model structure is needed for other end tasks (see Fig. 7). If the task input contains multiple sentences, a special delimiter token (\n$\n) is added between each pair of sentences. The embedding for this delimiter token is a new parameter we need to learn, but it should be pretty minimal.\nFor the sentence similarity task, because the ordering does not matter, both orderings are included. For the multiple choice task, the context is paired with every answer candidate.\nTraining objects in slightly modified GPT transformer models for downstream tasks. (Image source:\noriginal paper\n)\nSummary\n: It is super neat and encouraging to see that such a general framework is capable to beat SOTA on most language tasks at that time (June 2018). At the first stage, generative pre-training of a language model can absorb as much free text as possible. Then at the second stage, the model is fine-tuned on specific tasks with a small labeled dataset and a minimal set of new parameters to learn.\nOne limitation of GPT is its uni-directional nature — the model is only trained to predict the future left-to-right context.\nBERT\n#\nBERT\n, short for\nBidirectional Encoder Representations from Transformers\n(\nDevlin, et al., 2019\n) is a direct descendant to\nGPT\n: train a large language model on free text and then fine-tune on specific tasks without customized network architectures.\nCompared to GPT, the largest difference and improvement of BERT is to make training\nbi-directional\n. The model learns to predict both context on the left and right. The paper according to the ablation study claimed that:\n“bidirectional nature of our model is the single most important new contribution”\nPre-training Tasks\n#\nThe model architecture of BERT is a multi-layer bidirectional Transformer encoder.\nRecap of Transformer Encoder model architecture. (Image source:\nTransformer paper\n)\nTo encourage the bi-directional prediction and sentence-level understanding, BERT is trained with two tasks instead of the basic language task (that is, to predict the next token given context).\n*\nTask 1: Mask language model (MLM)\nFrom\nWikipedia\n: “A cloze test (also cloze deletion test) is an exercise, test, or assessment consisting of a portion of language with certain items, words, or signs removed (cloze text), where the participant is asked to replace the missing language item. … The exercise was first described by W.L. Taylor in 1953.”\nIt is unsurprising to believe that a representation that learns the context around a word rather than just after the word is able to better capture its meaning, both syntactically and semantically. BERT encourages the model to do so by training on the\n“mask language model” task\n:\nRandomly mask 15% of tokens in each sequence. Because if we only replace masked tokens with a special placeholder\n[MASK]\n, the special token would never be encountered during fine-tuning. Hence, BERT employed several heuristic tricks:\n(a) with 80% probability, replace the chosen words with\n[MASK]\n;\n(b) with 10% probability, replace with a random word;\n(c) with 10% probability, keep it the same.\nThe model only predicts the missing words, but it has no information on which words have been replaced or which words should be predicted. The output size is only 15% of the input size.\nTask 2: Next sentence prediction\nMotivated by the fact that many downstream tasks involve the understanding of relationships between sentences (i.e.,\nQA\n,\nNLI\n), BERT added another auxiliary task on training a\nbinary classifier\nfor telling whether one sentence is the next sentence of the other:\nSample sentence pairs (A, B) so that:\n(a) 50% of the time, B follows A;\n(b) 50% of the time, B does not follow A.\nThe model processes both sentences and output a binary label indicating whether B is the next sentence of A.\nThe training data for both auxiliary tasks above can be trivially generated from any monolingual corpus. Hence the scale of training is unbounded. The training loss is the sum of the mean masked LM likelihood and mean next sentence prediction likelihood.\nComparison of BERT, OpenAI GPT and ELMo model architectures. (Image source:\noriginal paper\n)\nInput Embedding\n#\nThe input embedding is the sum of three parts:\nWordPiece tokenization embeddings\n: The\nWordPiece\nmodel\nwas originally proposed for Japanese or Korean segmentation problem. Instead of using naturally split English word, they can be further divided into smaller sub-word units so that it is more effective to handle rare or unknown words. Please read\nlinked\npapers\nfor the optimal way to split words if interested.\nSegment embeddings\n: If the input contains two sentences, they have sentence A embeddings and sentence B embeddings respectively and they are separated by a special character\n[SEP]\n; Only sentence A embeddings are used if the input only contains one sentence.\nPosition embeddings\n: Positional embeddings are learned rather than hard-coded.\nBERT input representation. (Image source:\noriginal paper\n)\nNote that the first token is always forced to be\n[CLS]\n— a placeholder that will be used later for prediction in downstream tasks.\nUse BERT in Downstream Tasks\n#\nBERT fine-tuning requires only a few new parameters added, just like OpenAI GPT.\nFor classification tasks, we get the prediction by taking the final hidden state of the special first token\n[CLS]\n, $\\mathbf{h}^\\text{[CLS]}_L$, and multiplying it with a small weight matrix, $\\text{softmax}(\\mathbf{h}^\\text{[CLS]}_L \\mathbf{W}_\\text{cls})$.\nFor\nQA\ntasks like SQuAD, we need to predict the text span in the given paragraph for an given question. BERT predicts two probability distributions of every token, being the start and the end of the text span. Only two new small matrices, $\\mathbf{W}_\\text{s}$ and $\\mathbf{W}_\\text{e}$, are newly learned during fine-tuning and $\\text{softmax}(\\mathbf{h}^\\text{(i)}_L \\mathbf{W}_\\text{s})$ and $\\text{softmax}(\\mathbf{h}^\\text{(i)}_L \\mathbf{W}_\\text{e})$ define two probability distributions.\nOverall the add-on part for end task fine-tuning is very minimal — one or two weight matrices to convert the Transform hidden states to an interpretable format. Check the paper for implementation details for other cases.\nTraining objects in slightly modified BERT models for downstream tasks. (Image source:\noriginal paper\n)\nA summary table compares differences between fine-tuning of\nOpenAI GPT\nand BERT.\n| |\nOpenAI GPT\n|\nBERT\n|\n| Special char |\n[SEP]\nand\n[CLS]\nare only introduced at fine-tuning stage. |\n[SEP]\nand\n[CLS]\nand sentence A/B embeddings are learned at the pre-training stage. |\n| Training process | 1M steps, batch size 32k words. | 1M steps, batch size 128k words. |\n| Fine-tuning | lr = 5e-5 for all fine-tuning tasks. | Use task-specific lr for fine-tuning. |\nALBERT\n#\nALBERT\n(\nLan, et al. 2019\n), short for\nA Lite BERT\n, is a light-weighted version of\nBERT\nmodel. An ALBERT model can be trained 1.7x faster with 18x fewer parameters, compared to a BERT model of similar configuration. ALBERT incorporates three changes as follows: the first two help reduce parameters and memory consumption and hence speed up the training speed, while the third one proposes a more chanllenging training task to replace the next sentence prediction (NSP) objective.\nFactorized Embedding Parameterization\n#\nIn BERT, the WordPiece tokenization embedding size $E$ is configured to be the same as the hidden state size $H$. That is saying, if we want to increase the model size (larger $H$), we need to learn a larger tokenization embedding too, which is expensive because it depends on the vocabulary size ($V$).\nConceptually, because the tokenization embedding is expected to learn\ncontext-independent\nrepresentation and the hidden states are\ncontext-dependent\n, it makes sense to separate the size of the hidden layers from the size of vocabulary embedding. Using factorized embedding parameterization, the large vocabulary embedding matrix of size $V \\times H$ is decomposed into two small matrices of size $V \\times E$ and $E \\times H$. Given $H \\gt E$ or even $H \\gg E$, factorization can result in significant parameter reduction.\nCross-layer Parameter Sharing\n#\nParameter sharing across layers can happen in many ways: (a) only share feed-forward part; (b) only share attention parameters; or (c) share all the parameters. This technique reduces the number of parameters by a ton and does not damage the performance too much.\nSentence-Order Prediction (SOP)\n#\nInterestingly, the\nnext sentence prediction (NSP)\ntask of BERT turned out to be too easy. ALBERT instead adopted a sentence-order prediction (SOP)\nself-supervised\nloss,\nPositive sample: two consecutive segments from the same document.\nNegative sample: same as above, but the segment order is switched.\nFor the NSP task, the model can make reasonable predictions if it is able to detect topics when A and B are from different contexts. In comparison, SOP is harder as it requires the model to fully understand the coherence and ordering between segments.\nGPT-2\n#\nThe\nOpenAI\nGPT-2\nlanguage model is a direct successor to\nGPT\n. GPT-2 has 1.5B parameters, 10x more than the original GPT, and it achieves SOTA results on 7 out of 8 tested language modeling datasets in a\nzero-shot transfer setting\nwithout any task-specific fine-tuning. The pre-training dataset contains 8 million Web pages collected by crawling qualified outbound links from\nReddit\n. Large improvements by OpenAI GPT-2 are specially noticeable on small datasets and datasets used for measuring\nlong-term dependency\n.\nZero-Shot Transfer\n#\nThe pre-training task for GPT-2 is solely language modeling. All the downstream language tasks are framed as predicting conditional probabilities and there is no task-specific fine-tuning.\nText generation is straightforward using LM.\nMachine translation task, for example, English to Chinese, is induced by conditioning LM on pairs of “English sentence = Chinese sentence” and “the target English sentence =” at the end.\nFor example, the conditional probability to predict might look like:\nP(? | I like green apples. = 我喜欢绿苹果。 A cat meows at him. = 一只猫对他喵。It is raining cats and dogs. =\")\nQA task is formatted similar to translation with pairs of questions and answers in the context.\nSummarization task is induced by adding\nTL;DR:\nafter the articles in the context.\nBPE on Byte Sequences\n#\nSame as the original GPT, GPT-2 uses\nBPE\nbut on\nUTF-8\nbyte sequences. Each byte can represent 256 different values in 8 bits, while UTF-8 can use up to 4 bytes for one character, supporting up to $2^{31}$ characters in total. Therefore, with byte sequence representation we only need a vocabulary of size 256 and do not need to worry about pre-processing, tokenization, etc. Despite of the benefit, current byte-level LMs still have non-negligible performance gap with the SOTA word-level LMs.\nBPE merges frequently co-occurred byte pairs in a greedy manner. To prevent it from generating multiple versions of common words (i.e.\ndog.\n,\ndog!\nand\ndog?\nfor the word\ndog\n), GPT-2 prevents BPE from merging characters across categories (thus\ndog\nwould not be merged with punctuations like\n.\n,\n!\nand\n?\n). This tricks help increase the quality of the final byte segmentation.\nUsing the byte sequence representation, GPT-2 is able to assign a probability to any Unicode string, regardless of any pre-processing steps.\nModel Modifications\n#\nCompared to GPT, other than having many more transformer layers and parameters, GPT-2 incorporates only a few architecture modifications:\nLayer normalization\nwas moved to the input of each sub-block, similar to a residual unit of type\n“building block”\n(differently from the original type\n“bottleneck”\n, it has batch normalization applied before weight layers).\nAn additional layer normalization was added after the final self-attention block.\nA modified initialization was constructed as a function of the model depth.\nThe weights of residual layers were initially scaled by a factor of $1/ \\sqrt{N}$ where N is the number of residual layers.\nUse larger vocabulary size and context size.\nRoBERTa\n#\nRoBERTa\n(short for\nR\nobustly\no\nptimized\nBERT\na\npproach;\nLiu, et al. 2019\n) refers to a new receipt for training BERT to achieve better results, as they found that the original BERT model is significantly undertrained. The receipt contains the following learnings:\nTrain for longer with bigger batch size.\nRemove the\nnext sentence prediction (NSP)\ntask.\nUse longer sequences in training data format. The paper found that using individual sentences as inputs hurts downstream performance. Instead we should use multiple sentences sampled contiguously to form longer segments.\nChange the masking pattern dynamically. The original BERT applies masking once during the data preprocessing stage, resulting in a static mask across training epochs. RoBERTa applies masks in 10 different ways across 40 epochs.\nRoBERTa also added a new dataset\nCommonCrawl News\nand further confirmed that pretraining with\nmore data helps\nimprove the performance on downstream tasks. It was trained with the\nBPE on byte sequences\n, same as in\nGPT-2\n. They also found that choices of hyperparameters have a big impact on the model performance.\nT5\n#\nThe language model\nT5\nis short for\n“Text-to-Text Transfer Transformer”\n(\nRaffel et al., 2020\n). The encoder-decoder implementation follows the\noriginal Transformer\narchitecture: tokens → embedding → encoder → decoder → output. T5 adopts the framework “Natural Language Decathlon” (\nMcCann et al., 2018\n), where many common NLP tasks are translated into question-answering over a context. Instead of an explicit QA format, T5 uses short task prefixes to distinguish task intentions and separately fine-tunes the model on every individual task. The text-to-text framework enables easier transfer learning evaluation with the same model on a diverse set of tasks.\nA diagram of T5 task evaluation. The text-to-text framework casts every task into a generic form: feeding input text to predict some target text. (Image source:\nRaffel et al., 2020\n)\nThe model is trained on Web corpus extracted from Apr 2019 with various filters applied. The model is fine-tuned for each downstream task separately via “adapter layers” (add an extra layer for training) or “gradual unfreezing” (see\nULMFiT\n). Both fine-tuning approaches only update partial parameters while keeping the majority of the model parameters unchanged. T5-11B achieved SOTA results on many NLP tasks.\nAs the authors mentioned in the paper “…our goal is not to propose new methods but instead to provide a comprehensive perspective on where the field stands”, the T5 long paper described a lot of training setup and evaluation processes in detail, a good read for people who are interested in training a LM from scratch.\nGPT-3\n#\nGPT-3\n(\nBrown et al., 2020\n) has the same architecture as\nGPT-2\nbut contains 175B parameters, 10x larger than GPT-2 (1.5B). In addition, GPT-3 uses alternating dense and locally banded sparse attention patterns, same as in\nsparse transformer\n. In order to fit such a huge model across multiple GPUs, GPT-3 is trained with partitions along both width and depth dimension. The training data is a filtered version of Common Crawl mixed with a few other high-quality curated datasets. To avoid the contamination that downstream tasks might appear in the training data, the authors attempted to remove all the overlaps with all the studied benchmark dataset from the training dataset. Unfortunately the filtering process is not perfect due to a bug.\nTraining datasets for GPT-3. Note that the occurrence of each dataset during training is not proportional to the dataset size. (Table source:\nBrown et al., 2020\n)\nFor all the downstream evaluation, GPT-3 is tested in the few-shot setting without any gradient-based fine-tuning. Here the few-shot examples are provided as part of the prompt. GPT-3 achieves strong performance on many NLP datasets, comparable with fine-tuned BERT models.\nThe evaluation performance increases with the model size and the number of examples. (Image source:\nBrown et al., 2020\n)\nXLNet\n#\nThe\nAutoregressive (AR)\nmodel such as GPT and\nautoencoder (AE)\nmodel such as BERT are two most common ways for language modeling. However, each has their own disadvantages: AR does not learn the bidirectional context, which is needed by downstream tasks like reading comprehension and AE assumes masked positions are independent given all other unmasked tokens which oversimplifies the long context dependency.\nXLNet\n(\nYang et al. 2019\n) generalizes the AE method to incorporate the benefits of AR. XLNet proposed the\npermutation language modeling\nobjective. For a text sequence, it samples a factorization order $\\mathbf{z}$ and decomposes the likelihood $p_\\theta(\\mathbf{x})$ according to this factorization order,\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{XLNet} \n&= - \\mathbb{E}_{\\mathbf{z} \\sim \\mathcal{Z}_T} \\Big[ \\sum_{t=1}^T \\log p_\\theta (X_{z_t} = x \\mid \\mathbf{x}_{\\mathbf{z}_{<{t}}})\\Big] \\\\\n&= - \\mathbb{E}_{\\mathbf{z} \\sim \\mathcal{Z}_T} \\Big[ \\log \\frac{ \\exp(e(x)^\\top \\color{red}{h_\\theta (\\mathbf{x}_{\\mathbf{z}_{<{t}}})}) }{ \\sum_{x'} \\exp(e(x')^\\top \\color{red}{h_\\theta (\\mathbf{x}_{\\mathbf{z}_{<{t}}})}) } \\Big] \\\\\n&= - \\mathbb{E}_{\\mathbf{z} \\sim \\mathcal{Z}_T} \\Big[ \\log \\frac{ \\exp(e(x)^\\top \\color{blue}{g_\\theta (\\mathbf{x}_{\\mathbf{z}_{<{t}}}, z_t)}) }{ \\sum_{x'} \\exp(e(x')^\\top \\color{blue}{g_\\theta (\\mathbf{x}_{\\mathbf{z}_{<{t}}}, z_t)}) } \\Big]\n\\end{aligned}\n$$\nwhere $\\mathcal{Z}_T$ is a set of all possible permutation of length $T$; $z_t$ and $\\mathbf{z}_{