{ "source_file": "0930", "source_platform": "个人博客", "author": "Eugene Yan", "title_listed": "NLP for Supervised Learning - A Brief Survey", "keywords": "NLP,监督学习,综述", "domain": "文本处理", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://eugeneyan.com/writing/nlp-supervised-learning-survey/", "publish_date": "2020-06", "corpus_stage": "预训练/SFT", "notes": "NLP监督学习综述", "id": "b28b01614d51", "collected_at": "2026-10-02T14:17:06", "title": "NLP for Supervised Learning - A Brief Survey", "body_markdown": "eugeneyan\nStart Here\nWriting\nSpeaking\nPrototyping\nAbout\nNLP for Supervised Learning - A Brief Survey\n[\nllm\ndeeplearning\nsurvey\n]\n· 23 min read\nIt’s hard to keep up with the rapid progress of natural language processing (NLP). To organize my thoughts better, I took some time to review my notes, compare the various papers, and sort them chronologically. This helped in my understanding of how NLP (and its building blocks) has evolved over time.\nTo reinforce my learning, I’m writing this summary of the broad strokes, including brief explanations of how models work and some details (e.g., corpora, ablation studies). Here, we’ll see how NLP has progressed from 1985 till now:\nSequential models\n: RNN (1985), LSTM (1997), GRU (2014)\nWord embeddings\n: Word2vec (2013), GloVe (2014), FastText (2016)\nWord embeddings with context\n: ELMo (2018)\nAttention\n: Transformer (2017)\nPre-training\n: ULMFiT (2017), GPT (2017)\nCombining the above\n: BERT (2018)\nImproving BERT\n: DistilBERT, ALBERT, RoBERTa, XLNet (2019); Big Bird, Multilingual embeddings (2020)\nEverything is text-to-text\n: T5 (2019)\n(Did I miss anything important or oversimplify? Any errors? Please\nreach out\nwith suggestions and I’ll update. Thank you!)\nSequential models to process a sentences (1985)\nRecurrent neural networks (RNNs)\nwere first introduced around 1985 - 1986 (based on this\nthread\n). They differ from regular feedforward neural networks in that their hidden layers have connections to themselves. This allows them to operate over\nsequences\n(e.g., sentences of word tokens).\nThe state of the hidden layer at one time-step is used as input to the (same) hidden layer at the next time-step; thus the name “recurrent”. This allows the hidden layers to learn information about the temporal relationships between tokens in the sequence. For more details, check out Andrej Karpathy’s excellent\npost\n.\nRNN (left) and its unrolled view (right) (\nsource\n)\nHowever, RNNs had difficulty with modelling long-range dependencies (e.g., words that are far apart in a long sentence) due to vanishing/exploding gradients. With long sequences, the product of partial derivatives (through backpropagation) becomes very small (i.e., vanish). This happens when the partial derivatives are < 1. The reverse occurs when the partial derivatives are > 1 and the product explodes.\nThe\nLong Short-Term Memory (LSTM)\narchitecture introduced in 1997 improves on this. LSTMs model long-range dependencies better through\ngates\n:\nForget gate: Decides what information, from the current input and previous hidden state, to forget (i.e., discard) via a sigmoid\nInput gate: Decides what information to remember (i.e., stored) via a sigmoid\nOutput gate: Decides what the next hidden state should be\nTogether, these gates improve how the LSTM learns—what should be forgotten and what should be remembered? More details in Chris Olah’s\nexplanation\n. Though the LSTM was introduced in 1997, it wasn’t until 2015 that they saw commercial use:\nGoogle Voice\n, Apple’s\nQuickType\nand\nSiri\n, Amazon’s\nAlexa\n, and Facebook’s\nautomatic translations\n.\nComparison between the LSTM and GRU (\nsource\n)\nGated Recurrent Units (GRUs)\nsimplified the LSTM in 2014. It has only two gates, an update gate (similar to LSTM’s forget and input gate) and a reset gate (which also decides how much to forget). Because it has fewer gates (and thus fewer math operations), it’s faster to train. (In my work, I’ve found GRUs to converge faster with greater stability).\nWord embeddings to learn from unlabelled data (2013)\nIn 2013,\nWord2vec (w2v)\nwas introduced. Through\nunsupervised learning\n, it represents words as numbers, or more precisely, vectors of numbers. (Previously this was done via one-hot encoding). Being unsupervised, it’s able to learn on large corpora of unlabelled data (e.g., Wikipedia). When used in a variety of downstream tasks (e.g., classification), these word embeddings greatly improve model performance.\nThere are two ways to train w2v models:\nContinuous Bag of Words (CBOW)\nand\nSkip-gram\n. In CBOW, we predict the\ncenter\ntarget word given the context words around it. In Skip-gram, we predict the\nsurrounding\ncontext words given a center word (similar to CBOW, but in reverse). Skip-gram was found to work better with smaller amounts of data and to represent rare words better. CBOW trains faster and has better representations of more frequent words. (I’ve usually found Skip-gram to work better.)\nContinuous Bag of Words vs. Skip-gram (\nsource\n)\nWhy does Skip-gram work better for rare words?\nI've found this intuition helpful for understanding this. Let’s start with a sample sentence: “Today is a ___ day for writing”. The most common center words include “beautiful”, “productive”, “great” while rare words include “\nhellacious\n”.\nIn CBOW, the rare words (i.e, “hellacious”) are drowned out by the more frequent words. Thus, CBOW doesn’t learn representations of these rare words as well.\nHowever, with Skip-gram, these rare words don’t have to “compete” with the more common words (in terms of probability of occurrence). Instead, they have their own context pairs, such as (“is”, “hellacious”), (“a”, “hellacious”), (“hellacious”, “day”), and (“hellacious”, “for”). These context pairs are treated as separate observations and thus help Skip-gram to learn these rare words better.\nWord2vec applied\nsubsampling\n, where words that occurred relatively frequently were dropped out with a certain probability. This accelerated learning and improved word embeddings for rare words.\nIt also tweaked the problem slightly. Instead of predicting the most probable nearby words (out of all possible words), it tries to predict whether the word-pairs (from skip-gram) were actual pairs. This changes the final layer from a softmax with all the words (expensive) to a sigmoid that does binary classification (much cheaper) .\nHowever, our dataset and word pairs only have positive samples. Thus,\nnegative sampling\nis done to generate negative samples based on the distribution of the unigrams. (In the paper, they applied an interesting trick and raised the word counts to the 3/4 power.)\nGlobal Vectors for Word Representation (GloVe)\nwas introduced a year later (2014). While w2v learns word co-occurrence via a sliding window (i.e., local statistics), GloVe learns via a co-occurrence matrix (i.e., global statistics). GloVe then trains word vectors so their differences predict co-occurrence ratios. Surprisingly, though w2v and GloVe have different starting points, their word representations turn out to be similar.\nThere are also variants of w2v that learn subword or character embeddings. Subword embeddings learn the most frequent character segments (i.e., breaking a word into 2-3 character segments). One approach is\nFastText\n(where the original Word2vec author is a co-author). FastText open-sourced its\ncode\nas well as the\nmultiple-language word embeddings\ntrained with it.\nWhat did we do\nbefore\nembeddings?\nBefore embeddings, we mostly relied on n-grams and\none-hot vectors\nto represent text. (With embeddings, one-hot approaches are used less often. This occurs in non-NLP machine learning too, where categorical values are increasingly represented with embeddings instead of one-hot.)\nOn top of these one-hot vectors, we can apply\nsingular vector decomposition (SVD)\nfor topic modelling. Various supervised learning techniques (e.g.,\nlogistic regression, naive Bayes\n, decision trees, neural networks) can also be applied for classification (e.g., sentiment analysis, spam detection).\nAn example of this is the\nOtto Product Classification Competition\non Kaggle. In this competition, the dataset had 93 numerical features that represented counts of different events. This is assumed to be counts of product text tokens and other attributes. The top submissions used\nTF-IDF\nand ensembles of decision trees, K-nearest neighbours, naive Bayes, and neural networks (more\nhere\n).\nFor my\nproduct classification API\n, I adopted a combination of n-grams, TF-IDF, and naive Bayes. Paul Graham also used a similar approach for\nspam\ndetection\n.\nImproving word embeddings with context (2018)\nIn traditional word embeddings (e.g., Word2vec, GloVe), each token has only one representation (i.e., embedding). This is regardless of how it’s used in a sentence. For example, “date”:\nThey went out on a “date”.\nWhat “date” is it today?\nShe is eating her favourite fruit, a “date”.\nThat photo looks “dated”.\nEmbeddings from Language Models (ELMo)\nimproves on this (in 2018) by providing word representations\nbased on the entire sentence\n. It does this via a bidirectional language model (biLM). ELMO’s biLM comprises a two-layer bidirectional LSTM.\nBy going both left-to-right (LTR) and right-to-left (RTL), ELMo can learn more about a word’s context. For example, in “Today is a\nhellacious\nday for writing”, the LTR LM will encode “today is a” while the RTL LM will encode “writing for day”. Together, they consider the entire sentence for the word “hellacious”. These embeddings are learned via\nseparate\nLMs and concatenated before being used downstream. (Why do these LMs have to be\nseparate\n? See\nWhy can’t we use bidirectional context with multiple layers?\n)\nPre-trained ELMo can be used in a variety of supervised tasks. First, the biLM is trained and the word representation layers are frozen. Then, the ELMo word representation (i.e., vector) is concatenated with the token vector to enhance the word representation in the downstream task (e.g., classification).\nELMo embeddings concatenated with input tokens (the blue blocks) for downstream tasks (\nsource\n)\nAttention to remove the need for recurrence (2017)\n(We’re done with word representations; back to model architectures.)\nRecurrent models (e.g., RNN, LSTM, GRU) have a sequential nature—each hidden state requires the input of the previous hidden state. Thus, training cannot be parallelized. Furthermore, they can’t learn long-range dependencies well; while LSTM and GRU improved on the RNN, they too had their limits.\nThe\nTransformer\n(2017) solved both problems with\nattention\n. At a high level, attention determines how other tokens (in the input sequence) should be weighted while encoding the current token (i.e., how the others words in “Today is a hellacious day for writing” should weigh on “hellacious”). Together with the\npositional encodings\n(more later), we can process the entire sentence at once (no recurrence!) and compute each word’s representation based on the entire sequence.\nThe Transformer is made up of encoder and decoder stacks. In each encoder stack, there are six identical sub-layers, each having a self-attention mechanism followed by a fully connected feedforward neural network. The decoder stack is similar but includes an additional attention layer to learn attention over the encoder’s output. (It seems the Transformer was intended for seq2seq problems such as translation.)\nMulti-headed attention\n(eight heads) is used with each head randomly initialized. (Is it just me, or is this ensembling?) The outputs from these eight heads are concatenated and multiplied by an additional weight matrix. In the decoder stack, the attention mechanism is masked (to prevent looking ahead at future tokens).\nWhy\ndidn't\nthe chicken cross the road? Attention from two different heads (\nsource\n)\nTo provide information about the relative and absolute position of tokens in a sequence,\npositional encodings\nwere used at the bottom of the encoder and decoder stacks. This helps the model to learn from token positions as well as the distance between each token.\n(Be sure to check out Jay Alammar’s beautiful write-up on “\nThe Illustrated Transformer\n”.)\nFine-tuning learned embeddings (2017)\nSo far, we’ve mostly used word embeddings directly, or concatenated them with input tokens (i.e., ELMo). There was no fine-tuning of the word embeddings for specific tasks. This changed with ULMFiT and OpenAI GPT (\nand transfer learning\n).\nULMFiT\n(2017) uses\nAWD-LSTM\n(LSTM with dropout at the various gates) as its language model and introduced a fine-tuning phase as part of three steps. First, in\ngeneral-domain LM pre-training\n, the LM is trained on unlabelled data. ULMFiT trained on Wikitext-103 (28.6k Wikipedia articles and 103 million words).\nThen, in\ntarget-task fine-tuning\n, the LM is fine-tuned with the corpus of the target task (no labelled data introduced yet).\nDiscriminative fine-tuning\nis applied, where each layer is fine-tuned with different learning rates—the last layer has the highest learning rate, with each subsequent layer having reduced learning rates. ULMFiT also adopted\nslanted triangle learning rates\nwhere the learning rate increases quickly before decaying at a slower rate.\nFinally, in\ntarget-task classifier fine-tuning\n, two additional linear blocks are added on the LM (softmax for the last layer, ReLU for the intermediate layer). Gradual unfreezing is done, where we start with unfreezing the last LM layer and fine-tuning it. One by one, each subsequent layer is unfrozen and tuned.\nGPT\n(2017; granddaddy of\nGPT-3\n) also applied\nunsupervised pre-training\n. It uses the Transformer’s\ndecoder\nstack. This is an improvement over LSTMs as the Transformer has better learning on long-ranged dependencies and is not recurrent in nature.\nGPT is trained via two steps. First,\nunsupervised pre-training\n(similar to ULMFiT’s first step) involves learning on a corpus to predict the next word. GPT used the BookCorpus dataset of 7,000 unique, unpublished books. (This differs from ELMo, which uses shuffled sentences, thus destroying the long-range structure).\nThen,\nsupervised fine-tuning\ntweaks the decoder block for the target task. Task-specific inputs and labels are passed through the pre-trained decoder block to obtain the input representation (i.e., embedding). This representation is then fed into an additional linear output layer. In this stage, an auxiliary objective is included. The auxiliary objective predicts the next word in the task-specific corpus (similar to ULMFiT’s second step) and was found to improve generalisation and speed up convergence.\nInputs are transformed into a single sequence based on the task (\nsource\n)\nTo use GPT,\ntask-specific input transformations\nwere done for various tasks:\nText classification: The model can be used directly.\nTextual entailment: The premise and hypothesis is concatenated with a\n$\ntoken.\nSimilarity: We get both orders of the sentences (i.e., sentence1-sentence2, sentence2-sentence1) and concatenate them. The decoder output is added element-wise and fed into a linear layer.\nMultiple-choice Q&A: Concatenate the question with each possible answer and feed them through a linear layer. The outputs are normalised via softmax to get a probability distribution.\n(Check out Modern NLP’s\nTransfer Learning In NLP\nfor comprehensive review on this.)\nBERT: No recurrence, bidirectional, pre-trained (2018)\nTowards the end of 2018,\nBidirectional Encoder Representations from Transformers (BERT)\nwas introduced. It obtained SOTA results on eleven NLP tasks. There were several elements from previous models that made BERT a pragmatic SOTA model:\nTransformer: BERT uses the Transformer\nencoder\nstack and does away with recurrence (parallelizable training).\nELMo: Similar to ELMo, BERT has bidirectional context.\nULMFiT and GPT: BERT\nalso\nadopts pre-training and fine-tuning; thus, the LM (from pre-training) can be tuned for various downstream tasks.\nGPT: BERT also has a unified architecture and a single input representation.\nBERT uses a multi-layer bidirectional Transformer encoder block as its language model. In contrast, GPT uses a unidirectional Transformer decoder block while ELMo uses twin, separate LSTMs.\nInput to BERT is represented as a single sentence or a pair of sentences (e.g., question and answer). BERT uses\nWordPiece embeddings\n. It also introduces a special classification token (\nCLS\n) that is always the first token in a sequence—the final hidden state of this token is used for classification tasks. Sentence pairs are separated by a\nSEP\ntoken (similar to GPT).\nPre-training\nis done via two unsupervised tasks. First, a\nmasked language model (LM)\nis trained via the\ncloze\ntask. (The standard, predict-the-next-word task cannot be used with bidirectional context and multiple layers; more below). BERT masks 15% of tokens randomly (with the\nMASK\ntoken). However, this creates a mismatch between pre-training and fine-tuning. Thus, of the masked tokens, 80% are\nMASK\n, 10% are replaced by a random token, and 10% are unchanged. The LM predicts the original token with cross-entropy loss.\nWhy can’t we use bidirectional context with multiple layers?\nLet’s examine “Today is a hellacious day for writing” with attention layers.\nIn the first layer, the vector for “hellacious” does not include itself. However, in layer two, the vector of the words around “hellacious” will include the vector for “hellacious” due to attention (the orange arrow). (In the example below, I only show the this for “day”.) Thus, from layer 3 onwards, the vector for “hellacious” will be able to see itself as part of the weighted attention in layer 2 (the red arrow).\nHow tokens can see themselves via attention from lower layers\nThe second pre-training task involves\nnext sentence prediction (NSP)\n. Assuming two consecutive sentences A and B, 50% of the time sentence B actually follows sentence A. The other 50% of the time sentence B is a random sentence.\nFor both pre-training tasks, BERT uses the BooksCorpus (that GPT also used) and English Wikipedia (2,500 million words, text passages only). A document-level corpus was specifically chosen to learn from long contiguous sentences.\nFine-tuning\ninvolves passing task-specific inputs and labels to tweak model parameters end-to-end. Then, BERT can be used in various ways:\nSingle sentence and sentence-pair classification: Use the final hidden state of\nCLS\nSingle sentence tagging: Use the final hidden state of each token\nQ&A: Predict two probabilities per token, whether it’s the start or end of the text.\nBERT is practical (read: lower cost and faster to train due to no recurrence) and high performing. As a result, it has been\napplied to Google Search\n.\nMore examples of machine learning applied in the real-world:\napplied-ml\nAblation study: How does each BERT feature add to performance?\nAblation studies come from experimental neuropsychology, where parts of animals’ brains were removed to study the impact on their behaviour. In the context of machine learning, the term is used to describe removing certain parts of neural networks to gain a better understanding of the network behaviour.\nAblation studies are crucial for deep learning research -- can't stress this enough.\nUnderstanding causality in your system is the most straightforward way to generate reliable knowledge (the goal of any research). And ablation is a very low-effort way to look into causality.\n— François Chollet (@fchollet)\nJune 29, 2018\nFindings from ablation studies on BERT:\nRemoving next-sentence prediction reduced performance significantly.\nWithout bidirectional language modelling (i.e., only left-to-right like in GPT), performance is reduced significantly.\nA larger model often leads to accuracy improvements, even when the labelled training samples are as few as 3,600.\nUsing BERT for feature extraction (i.e., just using the word embeddings) also works well. The best approach is to concatenate the word representations from the top four hidden layers of the pre-trained encoder.\nWhile masked language modelling converges slower than left-to-right language modelling, masked language modelling eventually achieves higher accuracy with the same number of steps.\nMore details of the ablation studies in the BERT\npaper\n.\nImproving on BERT (2019+)\nSince BERT, several improvements have been made to make it lighter (\nDistilBERT\n,\nALBERT\n), optimize it further (\nRoBERTa\n). BERT has also been applied to create\nsentence embeddings in 109 languages\n.\nOne problem BERT had was not being able to learn bidirectionally (which is why BERT used the\ncloze\ntask for pre-training).\nXLNet\n(June 2019) addresses this via\npermutation language modelling (LM)\n; in contrast, BERT used masked language modelling.\nPermutation LM is tricky but here’s my attempt to explain it. Here’s our example sentence and token indices: “Today(1) is(2) a(3) hellacious(4) day(5) for(6) writing (7)”, with the sequence 1234567.\nFirst, permutation LM creates multiple sequence permutations (e.g., 2347516). Then, it learns representations of each word based on\nonly\nthe preceding tokens: token 4 learns from tokens 2 & 3; token 7 leads from tokens 2, 3, 4. This ensures that the predicted word is never seen indirectly. (Note: This permutation is only used in the computation of attention; the order of tokens in the sequence is unchanged.)\nBERT also had fixed-sized segments and thus could not handle sequences longer than the segment length (usually 512). XLNet uses previous work from\nTransformer-XL\nto get around this—it\ntransfers state across fixed-sized segments\n(i.e., recurrence on segments) to handle sequences longer than the segment length. Nonetheless, with permutation LM and reintroducing recurrence, XLNet is much more computationally demanding to train.\nRecently,\nBig Bird\n(28 July 2020) increased the segment length to 8x of what BERT could handle. BERT is limited by the\nquadratic\ndependency of its sequence length due to full attention, where each token has to attend to every other token. This leads to a memory limitation. On standard hardware (16gb RAM), this translates to input sequences (and segment length) of 512 tokens.\nBig Bird has sparser attention that allows it to model on 8x the sequence length of BERT (\nsource\n)\nBig Bird proposed\ngeneralized attention mechanism\nwhich has linear complexity (as opposed to quadratic) on the sequence length. It has three aspects of attention:\nRandom attention: Attends to\nr\nrandom tokens, leading to sparse attention (\nr\n= 2).\nSliding window attention: Attends to a window of width\nw\n(\nw\n= 3). This is similar to\nLongformer\nwhich uses a localised sliding window-based mask to reduce computation and extend BERT to longer sequences.\nGlobal tokens: Global tokens are a subset of\ng\ntokens in a sequence (\ng\n= 2). Global tokens attend to all tokens in the sequence and vice versa. This includes additional tokens such as\nCLS\nala BERT. (Using just random and sliding window attention were insufficient to compete with BERT).\nEverything is text-to-text\nTowards the end of 2019, the\nText-To-Text-Transfer-Transformer (T5)\nintroduced a\nunified framework that converts all text-based problems into a text-to-text format\n. Thus, the input and output are text-strings, making a single T5 fit for multiple tasks.\nTranslation, Q&A, classification are all text to text, with\ndifferent prefixes\nto indicate the task (\nsource\n)\nThe T5 uses the Transformer encoder-decoder based structure which the authors found to work best for text-to-text. Nonetheless, the authors introduced a few changes such as: removing the layer norm bias, placing the layer norm outside the residual path, and using a different position embedding scheme.\nPre-training was done on the\nColossal Clean Crawled Corpus (C4)\n, a high quality pre-processed English language corpus. It is approximately 750gb in size and is a cleaned version of the\nCommon Crawl\n.\nHow did C4 come about?\nC-4, or Composition C-4, was\ndeveloped by the British during World War II\n… I’m kidding. (Well, while that is how the explosive was developed, it’s not the C4 we’re discussing here. Context is important.)\nFirst, let’s start with the Common Crawl. The Common Crawl is a publicly available web archive of “web extracted text” from scraped HTML files. It produces\n20TB fo data each month\n. But most of the text is not natural language—there’s a lot of gibberish, boilerplate, error messages, etc.\nTo clean Common Crawl (and get C4, a 750gb dataset), the following was done:\nOnly retain lines ending with a terminal punctuation mark.\nDiscarded pages with <5 sentences that only retained lines with >= 3 words.\nRemoved pages with words on a “dirty, naughty, obscene, bad words” list.\nExcluded any line with the word\njavascript\n.\nRemoved pages with\nlorem\nipsum\n.\nRemoved pages with\n{\nto exclude pages containing code.\nDeduplicated three-sentence spans occurring more than once in the data.\nFiltered out pages not classified as English with probability of >= 0.99.\nWith the unified format, the authors thoroughly explored the\neffectiveness of transfer learning in NLP\n. I loved how a sizeable portion of the paper was devoted to experiments (and “What didn’t work”) on architectures, objectives, fine-tuning approaches, etc. Here’s a high-level summary (reading the original paper is recommended):\nThe encoder-decoder architecture with denoising objective worked best (vs. single Transformer stack, language modelling objective)\nAmong the unsupervised objectives, masked language modelling (BERT-style) worked best (vs. prefix language modelling, deshuffling, etc.)\nThere was limited difference between BERT-style objectives (e.g., replacing the entire corrupted span with a single\nMASK\n, dropping corrupted tokens entirely) and different corruption probabilities (e.g., 10%, 15%, 25%)\nTraining on the full dataset for 1 epoch worked better than training on a smaller data set for multiple epochs (my view:\nmore data > more compute\n)\nFine-tuning all layers simultaneously performed best (vs. unfreezing each layer and tuning it, adding adapter layers of various sizes)\nIncreasing model size had a bigger impact relative to increasing training time or batch size.\nHyperparameter tuning approach on unsupervised objectives for pre-training (\nsource\n)\nDownstream performance on a variety of benchmarks was evaluated (e.g., GLUE and SuperGLUE text classification, CNN/Daily Mail abstractive summarization, SQuAD question answering, and WMT translations.) With the insights from the experimental study, SOTA performance was achieved on 18 out of 24 tasks considered.\nSummary\nWhew, a lot has happened in NLP since 1985. To recap, here’s what we covered so far:\nSequential models\n: RNN (1985), LSTM (1997), GRU (2014)\nWord embeddings\n: Word2vec (2013), GloVe (2014), FastText (2016)\nWord embeddings with context\n: ELMo (2018)\nAttention\n: Transformer (2017)\nPre-training\n: ULMFiT (2017), GPT (2017)\nCombining the above\n: BERT (2018)\nImproving BERT\n: DistilBERT, ALBERT, RoBERTa, XLNet (2019); Big Bird, Multilingual embeddings (2020)\nEverything is text-to-text\n: T5 (2019)\nAnd here’s a table summary if that’s what you prefer.\nDetails\nRNN (1985)\nLearns on sequences; hidden state of one time-step used as input to next time-step\nLSTM (1997)\nAdds gates (e.g., forget, input) to RNN; models long-range dependencies better\nGRU (2014)\nSimplifies LSTM and has lesser gates (i.e., update, reset); faster to train\nWord2Vec (2013)\nUnsupervised learning of word representations via CBOW and Skip-gram with sampling and negative subsampling\nGloVe (2014)\nUnsupervised learning of word representations via co-occurrence matrix\nFastText (2016)\nUnsupervised learning of word representations via character segments\nELMo (2018)\nUnsupervised learning of word representations via bidirectional language model (two-layer LSTM); considers context of entire sentence\nTransformer (2017)\nRemoves need for recurrence via attention and positional encodings; faster and less compute heavy to train\nULMFiT (2017)\nPre-training and fine-tuning on AWD-LSTM with discriminative fine-tuning, slanted triangle learning rates, and gradual unfreezing\nGPT (2017)\nPre-training and fine-tuning on Transformer decoder stack; single input representation for various tasks\nBERT (2018)\nPre-training and fine-tuning on Transformer encoder stack, bidirectional context via masked language modelling, single input representation\nXLNet (2019)\nPermutation language modelling, recurrence across fixed sized segments to handle longer sequences\nT5 (2019)\nUnified text-to-text format on Transformer encoder-decoder architecture and denoising objective, trained on C4\nBig Bird (2020)\nGeneralized attention mechanism to increase segment length to 8x of BERT\nDid I miss any key milestones? Let me know by responding to this tweet or in the comments below!\nNLP is developing at a very fast pace.\nI recently took some time to review my notes and write a summary of the broad strokes (e.g., embeddings, architectures, transfer learning).\nDid I miss anything important/oversimplify? Any error? Feedback welcome!\nhttps://t.co/264TycIQDf\n— Eugene Yan (@eugeneyan)\nAugust 20, 2020\nReferences\nRNN:\nSerial Order: A Parallel Distributed Processing Approach\nLSTM:\nLong Short-Term Memory\nAWD-LSTM:\nRegularizing and Optimizing LSTM Language Models\nGRU:\nLearning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation\nWord2vec:\nEfficient Estimation of Word Representations in Vector Space\n,\nDistributed Representations of Words and Phrases and their Compositionality\nGloVe:\nGlobal Vectors for Word Representation\nFastText:\nEnriching Word Vectors with Subword Information\nELMo:\nDeep Contextualized Word Representations\nTransformer:\nAttention Is All You Need\nULMFiT:\nUniversal Language Model Fine-tuning for Text Classification\nGPT:\nImproving Language Understanding with Unsupervised Learning\nWordPiece embeddings:\nGoogle’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation\nBERT:\nPre-training of Deep Bidirectional Transformers for Language Understanding\nDistilBERT:\nSmaller, Faster, Cheaper, Lighter; A Distilled Version of BERT\nALBERT:\nA Lite BERT for Self-supervised Learning of Language Representations\nRoBERTa:\nA Robustly Optimized BERT Pretraining Approach\nTransformer-XL:\nAttentive Language Models Beyond a Fixed-Length Context\nMultilingual BERT embeddings:\nLanguage-agnostic BERT Sentence Embedding\nXLNet:\nGeneralized Autoregressive Pretraining for Language Understanding\nLongformer:\nThe Long-Document Transformer\nBig Bird:\nTransformers for Longer Sequences\nT5:\nExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer\nThanks\nto Yang Xinyi and\nPratik Bhavsar\nfor reading drafts of this.\nIf you found this useful, please cite this write-up as:\nYan, Ziyou. (Aug 2020). NLP for Supervised Learning - A Brief Survey. eugeneyan.com.\n https://eugeneyan.com/writing/nlp-supervised-learning-survey/.\nor\n@article{yan2020nlp,\n title = {NLP for Supervised Learning - A Brief Survey},\n author = {Yan, Ziyou},\n journal = {eugeneyan.com},\n year = {2020},\n month = {Aug},\n url = {https://eugeneyan.com/writing/nlp-supervised-learning-survey/}\n}\nShare on:\nBrowse related tags:\n[\nllm\ndeeplearning\nsurvey\n]\nor\nSearch\n« Unpopular Opinion: Data Scientists Should be More End-to-End\nEmbrace Beginner's Mind; Avoid The Wrong Way To Be An Expert »\nJoin\n11,800+\nreaders getting updates on machine learning, RecSys, LLMs, and engineering.\nGet email updates", "metadata": { "title": "NLP for Supervised Learning - A Brief Survey", "description": "Examining the broad strokes of NLP progress and comparing between models", "keywords": null, "author": "Eugene Yan", "og:site_name": "eugeneyan.com", "og:title": "NLP for Supervised Learning - A Brief Survey", "og:type": "article", "og:description": "Examining the broad strokes of NLP progress and comparing between models", "og:url": "https://eugeneyan.com/writing/nlp-supervised-learning-survey/", "og:image": "https://eugeneyan.com/assets/og_image/scrabble.jpg", "twitter:card": "summary_large_image", "twitter:domain": "eugeneyan.com", "twitter:title": "NLP for Supervised Learning - A Brief Survey", "twitter:description": "Examining the broad strokes of NLP progress and comparing between models", "twitter:image": "https://eugeneyan.com/assets/og_image/scrabble.jpg", "article:published_time": "2020-08-16T00:00:00+00:00", "article:author": "https://eugeneyan.com/about/", "article:section": "posts", "article:tag": "survey" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/b28b01614d51/code/body-000.txt", "chars": 252, "language": "text" } ], "images": [ { "status": "ok", "url": "https://eugeneyan.com/assets/icon-search.svg", "bytes": 425, "reused": false, "local_path": "attachments/b28b01614d51/images/body-feeb4bd87b8704a903b2.svg", "src": "https://eugeneyan.com/assets/icon-search.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/rnn.webp", "bytes": 15858, "reused": false, "local_path": "attachments/b28b01614d51/images/body-c085add9d68a60d8d02f.webp", "src": "https://eugeneyan.com/assets/rnn.webp", "alt": "RNN (left) and its unrolled view (right) ", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/lstm-gru.webp", "bytes": 15616, "reused": false, "local_path": "attachments/b28b01614d51/images/body-a0c541a6b9362cd64101.webp", "src": "https://eugeneyan.com/assets/lstm-gru.webp", "alt": "Comparison between the LSTM and GRU", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/cbow-vs-skipgram.webp", "bytes": 9632, "reused": false, "local_path": "attachments/b28b01614d51/images/body-2dd9942aa80852de3111.webp", "src": "https://eugeneyan.com/assets/cbow-vs-skipgram.webp", "alt": "Continuous Bag of Words vs. Skip-gram", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/elmo.webp", "bytes": 19064, "reused": false, "local_path": "attachments/b28b01614d51/images/body-f3fcca9f2620ac6511dc.webp", "src": "https://eugeneyan.com/assets/elmo.webp", "alt": "ELMo embeddings are concatenated with input tokens (the blue blocks) for downstream tasks", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/attention.webp", "bytes": 15770, "reused": false, "local_path": "attachments/b28b01614d51/images/body-34e31d8ff45335f79299.webp", "src": "https://eugeneyan.com/assets/attention.webp", "alt": "Attention from two different heads", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/gpt-input.webp", "bytes": 33128, "reused": false, "local_path": "attachments/b28b01614d51/images/body-fc30b4dd4302de5deac7.webp", "src": "https://eugeneyan.com/assets/gpt-input.webp", "alt": "Inputs are transformed into a single sequence based on the task", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/bidirectional-attention.webp", "bytes": 11994, "reused": false, "local_path": "attachments/b28b01614d51/images/body-439ebfaf01e706c6c3ed.webp", "src": "https://eugeneyan.com/assets/bidirectional-attention.webp", "alt": "How tokens can see themselves via attention from lower layers", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/bigbird.webp", "bytes": 21380, "reused": false, "local_path": "attachments/b28b01614d51/images/body-5189c5513e20e420354d.webp", "src": "https://eugeneyan.com/assets/bigbird.webp", "alt": "Big Bird has sparser attention that allows it to model on 8x the sequence length of BERT", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/text-to-text.webp", "bytes": 17538, "reused": false, "local_path": "attachments/b28b01614d51/images/body-0acf8665e1ef88e96be7.webp", "src": "https://eugeneyan.com/assets/text-to-text.webp", "alt": "Translation, Q&A, classification are all text to text, with different prefixes to indicate the task", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/unsupervised-objectives.webp", "bytes": 17610, "reused": false, "local_path": "attachments/b28b01614d51/images/body-3b9d63f4c76e2749e05d.webp", "src": "https://eugeneyan.com/assets/unsupervised-objectives.webp", "alt": "Hyperparameter tuning approach on unsupervised objectives for pre-training", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/icon-twitter.svg", "bytes": 892, "reused": false, "local_path": "attachments/b28b01614d51/images/body-0083a7581b07e5f40e7b.svg", "src": "https://eugeneyan.com/assets/icon-twitter.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/icon-linkedin.svg", "bytes": 559, "reused": false, "local_path": "attachments/b28b01614d51/images/body-f7a16c3e46e5838cec4d.svg", "src": "https://eugeneyan.com/assets/icon-linkedin.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/bluesky.svg", "bytes": 745, "reused": false, "local_path": "attachments/b28b01614d51/images/body-be754e8ecdaf6de94f55.svg", "src": "https://eugeneyan.com/assets/bluesky.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/icon-facebook.svg", "bytes": 1123, "reused": false, "local_path": "attachments/b28b01614d51/images/body-5d55b59733923cf14223.svg", "src": "https://eugeneyan.com/assets/icon-facebook.svg", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://eugeneyan.com/assets/icon-mail.svg", "bytes": 841, "reused": false, "local_path": "attachments/b28b01614d51/images/body-b179147617228dfafb9f.svg", "src": "https://eugeneyan.com/assets/icon-mail.svg", "alt": "", "downloaded": true } ], "audio": [], "videos": [ { "url": "https://www.youtube.com/watch?v=tG3pUwmGjsc", "src": "https://www.youtube.com/watch?v=tG3pUwmGjsc", "status": "ok", "kind": "single_video", "local_path": "attachments/b28b01614d51/videos/repair-tG3pUwmGjsc.mp4", "bytes": 386216694, "valid": true, "height": 1080, "width": 1920, "duration": "3999.261000", "reused": false }, { "url": "https://www.youtube.com/watch?v=dt7sArnLo1g", "src": "https://www.youtube.com/watch?v=dt7sArnLo1g", "status": "ok", "kind": "single_video", "local_path": "attachments/b28b01614d51/videos/repair-dt7sArnLo1g.mp4", "bytes": 237551325, "valid": true, "height": 1080, "width": 1920, "duration": "3088.521000", "reused": false } ], "links": [ { "href": "https://eugeneyan.com/", "label": "eugeneyan", "type": "a" }, { "href": "https://eugeneyan.com/start-here/", "label": "Start Here", "type": "a" }, { "href": "https://eugeneyan.com/writing/", "label": "Writing", "type": "a" }, { "href": "https://eugeneyan.com/speaking/", "label": "Speaking", "type": "a" }, { "href": "https://eugeneyan.com/prototyping/", "label": "Prototyping", "type": "a" }, { "href": "https://eugeneyan.com/about/", "label": "About", "type": "a" }, { "href": "https://eugeneyan.com/search/", "label": "", "type": "a" }, { "href": "https://eugeneyan.com/tag/llm/", "label": "llm", "type": "a" }, { "href": "https://eugeneyan.com/tag/deeplearning/", "label": "deeplearning", "type": "a" }, { "href": "https://eugeneyan.com/tag/survey/", "label": "survey", "type": "a" }, { "href": "https://twitter.com/eugeneyan", "label": "reach out", "type": "a" }, { "href": "http://cseweb.ucsd.edu/~gary/PAPER-SUGGESTIONS/Jordan-TR-8604-OCRed.pdf", "label": "Recurrent neural networks (RNNs)", "type": "a" }, { "href": "https://ai.stackexchange.com/questions/8190/where-can-i-find-the-original-paper-that-introduced-rnns", "label": "thread", "type": "a" }, { "href": "http://karpathy.github.io/2015/05/21/rnn-effectiveness/", "label": "post", "type": "a" }, { "href": "https://brilliant.org/wiki/recurrent-neural-network/", "label": "source", "type": "a" }, { "href": "https://dl.acm.org/doi/10.1162/neco.1997.9.8.1735", "label": "Long Short-Term Memory (LSTM)", "type": "a" }, { "href": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/", "label": "explanation", "type": "a" }, { "href": "https://ai.googleblog.com/2015/08/the-neural-networks-behind-google-voice.html", "label": "Google Voice", "type": "a" }, { "href": "https://www.theinformation.com/articles/apples-machines-can-learn-too", "label": "QuickType", "type": "a" }, { "href": "https://bgr.com/2016/06/13/ios-10-siri-third-party-apps/", "label": "Siri", "type": "a" }, { "href": "http://www.allthingsdistributed.com/2016/11/amazon-ai-and-alexa-for-all-aws-apps.html", "label": "Alexa", "type": "a" }, { "href": "https://www.theverge.com/2017/8/4/16093872/facebook-ai-translations-artificial-intelligence", "label": "automatic translations", "type": "a" }, { "href": "https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21", "label": "source", "type": "a" }, { "href": "https://arxiv.org/abs/1406.1078", "label": "Gated Recurrent Units (GRUs)", "type": "a" }, { "href": "https://arxiv.org/abs/1301.3781", "label": "Word2vec (w2v)", "type": "a" }, { "href": "https://arxiv.org/pdf/1301.3781.pdf", "label": "source", "type": "a" }, { "href": "https://www.merriam-webster.com/dictionary/hellacious", "label": "hellacious", "type": "a" }, { "href": "https://nlp.stanford.edu/projects/glove/", "label": "Global Vectors for Word Representation (GloVe)", "type": "a" }, { "href": "https://arxiv.org/abs/1607.04606", "label": "FastText", "type": "a" }, { "href": "https://github.com/facebookresearch/fastText", "label": "code", "type": "a" }, { "href": "https://fasttext.cc/", "label": "multiple-language word embeddings", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/One-hot", "label": "one-hot vectors", "type": "a" }, { "href": "https://www.youtube.com/watch?v=tG3pUwmGjsc", "label": "singular vector decomposition (SVD)", "type": "a" }, { "href": "https://www.youtube.com/watch?v=dt7sArnLo1g", "label": "logistic regression, naive Bayes", "type": "a" }, { "href": "https://www.kaggle.com/c/otto-group-product-classification-challenge", "label": "Otto Product Classification Competition", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Tf%E2%80%93idf", "label": "TF-IDF", "type": "a" }, { "href": "https://www.slideshare.net/eugeneyan/kaggle-otto-challenge-how-we-achieved-85th-out-of-3845-and-what-we", "label": "here", "type": "a" }, { "href": "https://eugeneyan.com/writing/product-categorization-api-part-3-creating-an-api/", "label": "product classification API", "type": "a" }, { "href": "http://paulgraham.com/spam.html", "label": "spam", "type": "a" }, { "href": "http://paulgraham.com/better.html", "label": "detection", "type": "a" }, { "href": "https://arxiv.org/abs/1802.05365", "label": "Embeddings from Language Models (ELMo)", "type": "a" }, { "href": "https://www.slideshare.net/shuntaroy/a-review-of-deep-contextualized-word-representations-peters-2018", "label": "source", "type": "a" }, { "href": "https://arxiv.org/abs/1706.03762", "label": "Transformer", "type": "a" }, { "href": "https://colab.research.google.com/github/tensorflow/tensor2tensor/blob/master/tensor2tensor/notebooks/hello_t2t.ipynb", "label": "source", "type": "a" }, { "href": "https://jalammar.github.io/illustrated-transformer/", "label": "The Illustrated Transformer", "type": "a" }, { "href": "https://arxiv.org/abs/1801.06146", "label": "ULMFiT", "type": "a" }, { "href": "https://arxiv.org/abs/1708.02182", "label": "AWD-LSTM", "type": "a" }, { "href": "https://openai.com/blog/language-unsupervised/", "label": "GPT", "type": "a" }, { "href": "https://openai.com/blog/openai-api/", "label": "GPT-3", "type": "a" }, { "href": "https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf", "label": "source", "type": "a" }, { "href": "https://medium.com/modern-nlp/transfer-learning-in-nlp-f5035cc3f62f", "label": "Transfer Learning In NLP", "type": "a" }, { "href": "https://arxiv.org/abs/1810.04805", "label": "Bidirectional Encoder Representations from Transformers (BERT)", "type": "a" }, { "href": "https://arxiv.org/abs/1609.08144", "label": "WordPiece embeddings", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Cloze_test", "label": "cloze", "type": "a" }, { "href": "https://www.blog.google/products/search/search-language-understanding-bert/", "label": "applied to Google Search", "type": "a" }, { "href": "https://github.com/eugeneyan/applied-ml", "label": "applied-ml", "type": "a" }, { "href": "https://twitter.com/fchollet/status/1012721582148550662?ref_src=twsrc%5Etfw", "label": "June 29, 2018", "type": "a" }, { "href": "https://medium.com/huggingface/distilbert-8cf3380435b5", "label": "DistilBERT", "type": "a" }, { "href": "https://arxiv.org/abs/1909.11942", "label": "ALBERT", "type": "a" }, { "href": "https://arxiv.org/abs/1907.11692", "label": "RoBERTa", "type": "a" }, { "href": "https://arxiv.org/abs/2007.01852", "label": "sentence embeddings in 109 languages", "type": "a" }, { "href": "https://arxiv.org/abs/1906.08237", "label": "XLNet", "type": "a" }, { "href": "https://arxiv.org/abs/1901.02860", "label": "Transformer-XL", "type": "a" }, { "href": "https://arxiv.org/abs/2007.14062", "label": "Big Bird", "type": "a" }, { "href": "https://arxiv.org/abs/2004.05150", "label": "Longformer", "type": "a" }, { "href": "https://arxiv.org/pdf/1910.10683.pdf", "label": "Text-To-Text-Transfer-Transformer (T5)", "type": "a" }, { "href": "https://www.tensorflow.org/datasets/catalog/c4", "label": "Colossal Clean Crawled Corpus (C4)", "type": "a" }, { "href": "https://commoncrawl.org/", "label": "Common Crawl", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/C-4_(explosive)", "label": "developed by the British during World War II", "type": "a" }, { "href": "https://t.co/264TycIQDf", "label": "https://t.co/264TycIQDf", "type": "a" }, { "href": "https://twitter.com/eugeneyan/status/1296236611902951424?ref_src=twsrc%5Etfw", "label": "August 20, 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1310.4546", "label": "Distributed Representations of Words and Phrases and their Compositionality", "type": "a" }, { "href": "https://arxiv.org/abs/1910.10683", "label": "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", "type": "a" }, { "href": "https://twitter.com/nlpguy_", "label": "Pratik Bhavsar", "type": "a" }, { "href": "https://eugeneyan.com/writing/end-to-end-data-science/", "label": "« Unpopular Opinion: Data Scientists Should be More End-to-End", "type": "a" }, { "href": "https://eugeneyan.com/writing/beginners-mind/", "label": "Embrace Beginner's Mind; Avoid The Wrong Way To Be An Expert »", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:52:22Z", "source_url": "https://eugeneyan.com/writing/nlp-supervised-learning-survey/", "papers": [ { "status": "ok", "url": "http://cseweb.ucsd.edu/~gary/PAPER-SUGGESTIONS/Jordan-TR-8604-OCRed.pdf", "bytes": 2281776, "reused": false, "local_path": "papers/repair/6cd908c4d5e418ca93b3.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1406.1078", "bytes": 1142296, "reused": false, "local_path": "papers/repair/7d6f882033801fb7380b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1301.3781", "bytes": 228716, "reused": true, "local_path": "papers/repair/56b49aeaee8219bc1cd4.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1301.3781.pdf", "bytes": 228716, "reused": true, "local_path": "papers/repair/1e8e84311c23b0346b07.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1607.04606", "bytes": 2374847, "reused": true, "local_path": "papers/repair/f1838fe5efe89dedd7be.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1802.05365", "bytes": 425880, "reused": true, "local_path": "papers/repair/c4dc3d740cad43df1dd2.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1706.03762", "bytes": 2215244, "reused": true, "local_path": "papers/repair/c3ab01e47e614fe3cf33.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1801.06146", "bytes": 979417, "reused": true, "local_path": "papers/repair/f99f602eebf2965d0ed0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1708.02182", "bytes": 185413, "reused": false, "local_path": "papers/repair/f2555bb43fcea5781c7c.pdf" }, { "status": "ok", "url": "https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf", "bytes": 541036, "reused": true, "local_path": "papers/repair/a3e86ad05f37e40d21b6.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1810.04805", "bytes": 775166, "reused": true, "local_path": "papers/repair/33769e0bafc9f0c73baa.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1609.08144", "bytes": 1688289, "reused": true, "local_path": "papers/repair/f8e7010861b5c08375c8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.11942", "bytes": 419105, "reused": true, "local_path": "papers/repair/9c6f91591bd290ea5163.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1907.11692", "bytes": 209675, "reused": true, "local_path": "papers/repair/8acd467c11db68744d3f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2007.01852", "bytes": 695757, "reused": false, "local_path": "papers/repair/26ae0d7e6b615d651b90.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1906.08237", "bytes": 761790, "reused": false, "local_path": "papers/repair/e1dc08bfcdc2d830fa23.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1901.02860", "bytes": 4570910, "reused": true, "local_path": "papers/repair/8514ec65c2953ee68ccf.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2007.14062", "bytes": 1153823, "reused": true, "local_path": "papers/repair/f56d7d4a1317893ab3ad.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2004.05150", "bytes": 539211, "reused": false, "local_path": "papers/repair/e1ed8e9d5c91ac41f458.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1910.10683.pdf", "bytes": 1190021, "reused": false, "local_path": "papers/repair/a7cbb6be7ac1a888a45f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1310.4546", "bytes": 125222, "reused": false, "local_path": "papers/repair/ea4c9ced9df7306b4efc.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1910.10683", "bytes": 1190021, "reused": true, "local_path": "papers/repair/4e0745eb290124ed9e02.pdf" } ], "actual_video_files": [], "body_chars": 30986, "expected_body_images": 16, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/b28b01614d51/pages/source.html", "local_html": "attachments/b28b01614d51/pages/article.html", "body_base_path": "attachments/b28b01614d51/pages/", "body_markdown_path": "attachments/b28b01614d51/pages/article.md" }