{ "updated_at": "2026-10-20T22:34:39Z", "series": [ { "source_excel_row": 132, "source_url": "https://nlp.elvissaravia.com/p/ai-agents-weekly", "items": [ { "url": "https://nlp.elvissaravia.com/p/ai-agents-weekly", "status": "http_404" } ], "updated_at": "2026-10-14T11:43:08Z", "finished": true, "remaining": 0, "repair_version": 3 }, { "source_excel_row": 1602, "source_url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/", "items": [ { "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/", "status": "ok", "article_id": "1cc4df7eb2c7", "local_path": "series-follow/1602/repair/caf69956cb5cfd00e2de.html", "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/EDA-exp1.png", "bytes": 299081, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/b1e26be047c763592a3a.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GPT3-data-gen.png", "bytes": 178022, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/364dc68a08807b5baf8a.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GPT3-data-gen-exp.png", "bytes": 979053, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/2a8613e8ff2f07c9bbde.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/LAMBADA.png", "bytes": 163034, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/1e49c821edae4f24f95b.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/UDG.png", "bytes": 396874, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/773377f93dad7a113410.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/UDG-exp.png", "bytes": 103983, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/82e84409bcacc2fc600e.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/back-translation-data-gen.png", "bytes": 161004, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/3800c7c1476957d08f84.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/back-translation-data-gen-exp.png", "bytes": 107466, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/9ca541330b72888db6f9.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/affinity-diversity.png", "bytes": 257673, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/6945c914870c32780b47.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/noise-adaptation-layer.png", "bytes": 193590, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/1e20893406355a07939d.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GLC.png", "bytes": 283685, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/6185b7cd92536235d8de.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/L2R-backprop.png", "bytes": 241069, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/67e6e4a22214ef58e33c.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/L2R-exp.png", "bytes": 812915, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/7d52e35a3da415faf3df.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/MentorNet.png", "bytes": 260104, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/e2db6011ba9c735f3718.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/co-teaching.png", "bytes": 313297, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/cb3814a9fb4722b3289a.png" } ], "article": { "source_file": "0930", "source_platform": "个人博客", "author": "Lilian Weng / Lil'Log", "title_listed": "Learning with not Enough Data Part 3: Data Generation", "keywords": "数据增强,合成数据,生成", "domain": "数据处理", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/", "publish_date": "2022-04", "corpus_stage": "数据处理", "notes": "数据生成系列综述之三", "id": "1cc4df7eb2c7", "collected_at": "2026-10-02T14:13:06", "title": "Learning with not Enough Data Part 3: Data Generation | Lil'Log", "body_markdown": "Here comes the Part 3 on learning with not enough data (Previous:\nPart 1\nand\nPart 2\n). Let’s consider two approaches for generating synthetic data for training.\nAugmented data\n. Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a\nprevious post\non contrastive learning. For the sake of post completeness, I\nduplicate\nthe section on data augmentation here with some edits.\nNew data\n. Given few or even no data points, we can rely on powerful pretrained models to generate a number of\nnew\ndata points. This is especially true in recent years given the fast progress in large pretrained\nlanguage models (LM)\n. Few shot prompting is shown to be effective for LM to learn within context without extra training.\nData Augmentation\n#\nThe goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.\nImage Augmentation\n#\nBasic Image Processing Operations\n#\nThere are several ways to modify an image while retaining its semantic information. We can use any one of the following augmentation or a composition of multiple operations.\nRandom cropping and then resize back to the original size.\nRandom color distortions\nRandom Gaussian blur\nRandom color jittering\nRandom horizontal flip\nRandom grayscale conversion\nAnd many more. Check\nPIL.ImageOps\nfor inspiration.\nTask-Specific Augmentation Strategies\n#\nIf the downstream task is known, it is possible to learn the optimal augmentation strategies (i.e. what processing operations to use and how to combine them in sequence) to maximize the downstream task performance.\nAutoAugment\n(\nCubuk, et al. 2018\n) is inspired by\nneural architecture search\n, AutoAugment frames the problem of learning best data augmentation operations (i.e. shearing, rotation, invert, etc.) for image classification as an RL problem and looks for the combination that leads to the highest accuracy on the evaluation set. AutoAugment can be executed in adversarial fashion (\nZhang, et al 2019\n).\nRandAugment\n(\nCubuk et al., 2019\n) greatly reduces the search space of AutoAugment by controlling the magnitudes of different transformation operations with a single magnitude parameter.\nPopulation based augmentation\n(PBA;\nHo et al., 2019\n) combines PBT (“population based training”;\nJaderberg et al, 2017\n) with AutoAugment, using the evolutionary algorithm to train a population of children models in parallel to evolve the best augmentation strategies.\nUnsupervised Data Augmentation\n(UDA;\nXie et al., 2019\n), among a set of possible augmentation strategies, selects a subset to minimize the KL divergence between the predicted distribution over an unlabelled example and its unlabelled augmented version.\nImage Mixture\n#\nImage mixture methods can construct new training examples from existing data points.\nMixup\n(\nZhang et al., 2018\n) runs global-level mixture by creating a weighted pixel-wise combination of two existing images $I_1$ and $I_2$: $I_\\text{mixup} \\gets \\alpha I_1 + (1-\\alpha) I_2$ and $\\alpha \\in [0, 1]$.\nCutmix\n(\nYun et al., 2019\n) does region-level mixture by generating a new example by combining a local region of one image with the rest of the other image. $I_\\text{cutmix} \\gets \\mathbf{M}_b \\odot I_1 + (1-\\mathbf{M}_b) \\odot I_2$, where $\\mathbf{M}_b \\in \\{0, 1\\}^I$ is a binary mask and $\\odot$ is element-wise multiplication. It is equivalent to filling the\ncutout\n(\nDeVries & Taylor 2017\n) region with the same region from another image.\nGiven a query $\\mathbf{q}$,\nMoCHi\n(“mixing of contrastive hard negatives”;\nKalantidis et al. 2020\n) maintains a queue of $K$ negative features $Q={\\mathbf{n}_1, \\dots, \\mathbf{n}_K }$ and sorts these negative features by similarity to the query, $\\mathbf{q}^\\top \\mathbf{n}$, in descending order. The first $N$ items in the queue are considered as the hardest negatives, $Q^N$. Then synthetic hard examples can be generated by $\\mathbf{h} = \\tilde{\\mathbf{h}} / |\\tilde{\\mathbf{h}}|_2$ where $\\tilde{\\mathbf{h}} = \\alpha\\mathbf{n}_i + (1-\\alpha) \\mathbf{n}_j$ and $\\alpha \\in (0, 1)$. Even harder examples can be created by mixing with the query feature, $\\mathbf{h}’ = \\tilde{\\mathbf{h}’} / |\\tilde{\\mathbf{h}’}|_2$ where $\\tilde{\\mathbf{h}’} = \\beta\\mathbf{q} + (1-\\beta) \\mathbf{n}_j$ and $\\beta \\in (0, 0.5)$.\nText Augmentation\n#\nLexical Edits\n#\nEasy Data Augmentation\n(EDA;\nWei & Zou 2019\n) defines a set of simple but powerful operations for text augmentation. Given a sentence, EDA randomly chooses and applies one of four simple operations:\nSynonym replacement (SR): Replace $n$ random non-stop words with their synonyms.\nRandom insertion (RI): Place a random synonym of a randomly selected non-stop word in the sentence at a random position.\nRandom swap (RS): Randomly swap two words and repeat $n$ times.\nRandom deletion (RD): Randomly delete each word in the sentence with probability $p$.\nwhere $p=\\alpha$ and $n=\\alpha \\times \\text{sentence_length}$, with the intuition that longer sentences can absorb more noise while maintaining the original label. The hyperparameter $\\alpha$ roughly indicates the percent of words in one sentence that may be changed by one augmentation.\nEDA is shown to improve the classification accuracy on several classification benchmark datasets compared to baseline without EDA. The performance lift is more significant on a\nsmaller\ntraining set. All the four operations in EDA help improve the classification accuracy, but get to optimal at different $\\alpha$’s.\nEDA leads to performance improvement on several classification benchmarks. (Image source:\nWei & Zou 2019\n)\nContextual\nAugmentation (\nKobayashi, 2018\n) replaces word $w_i$ at position $i$ by sampling from a probability distribution learned by a bidirectional LM such as BERT, $p(.\\mid S\\setminus{w_i})$. In this way, the words are substituted by synonyms, or similar words suitable for the context. To guarantee such operations do not alter the labels, the LM is fit to be label-conditioned bidirectional LM. Conditional BERT (CBERT;\nXing Wu et al. 2018\n) extends BERT to predict masked tokens conditioned on the class label and can be used for contextual augmentation prediction.\nBack-translation\n#\nBack-translation\nproduces augmented data by translating text samples to another language and then translating them back. The translation happens in two ways and both directions should have decent enough performance to avoid significant loss of semantic meaning.\nMix-up\n#\nIt is also possible to apply\nMixup\nto text (\nGuo et al. 2019\n) but on the embedding space to obtain some performance gain. The proposed method relies on a specially designed model architecture to operate the prediction on the word or sentence embedding. Adding adversarial noise in the embedding space as a way of data augmentation is shown to improve the generalization of model training (\nZhu et al. 2019\n).\nAudio Augmentation\n#\nHere is a list of several commonly used audio data augmentation methods, operated on raw audio or spectrograms, summarized by\nWang & van den Oord (2021)\n.\nAudio mixup.\nGiven two audio clips $\\mathbf{x}_1$ and $\\mathbf{x}_2$, the mixed-up version $\\hat{\\mathbf{x}} = \\alpha \\mathbf{x}_1 + (1-\\alpha)\\mathbf{x}_2$ should be associated with the label of the more dominant input. The audio mixup augments the data with more realistic noise.\nTime masking.\nA small consecutive chunk of the audio can be masked without losing semantic information.\nFrequency masking.\nA small amount of frequency components on the spectrogram can be dropped off and it should not change the associated label.\nFrequency shift.\nThe spectrogram can be shifted by an integer between $[-F, F]$, where $F$ is the maximum shift size. It is a cheap augmentation to change the pitch of the audio.\nArchitectural Augmentation\n#\nModels with\ndropout\nlayers can create augmented samples by applying different dropout masks on the same input sample. For example, in the contrastive learning model\nSimCSE\n(\nGuo et al. 2021\n), a sample is simply fed into the encoder twice with different dropout masks and these two versions are the positive pair where the other in-batch samples are considered as negative pairs.\nDropout augments data by adding noise onto the internal representation of the model. It can be applied in a more structured way, such as in\ncutoff\n(\nShen et al. (2020)\n), where random chunks of the token embedding matrix are removed.\nData Synthesis\n#\nGiven that generating high-quality, photorealistic images is a lot more difficult than generating human-like natural language text and recent success with large pretrained language models, this section only focuses on text generation. To read more on how to synthesize realistic images, check posts on\nGAN\n,\nVAE\n,\nflow\nand\ndiffusion\nmodels.\nLanguage Model as Noisy Annotator\n#\nWang et al. (2021)\nexplored ways to leverage GPT-3 as a weak annotator via few-shot prompting, achieving 10x cheaper than human labeling. The paper argues that by using data labeled by GPT-3, it essentially performs\nself-training\n: The predictions on unlabeled samples apply entropy regularization on the model to avoid high class overlaps so as to help improve the model performance.\nIllustration of how to use GPT-3 to generate more training data with the human-in-the-loop active learning pipeline to improve the data quality. (Image source:\nWang et al. 2021\n)\nGPT-3-labeled samples selected by\nactive learning\nwith highest uncertainty are sent to human labelers to be re-annotated. The few-shot prompt contains a small number of human labeled examples and thus the labeling cost is restricted. Synthetic samples are ranked by predicted logits of label $y$ and those with the lowest scores go through relabeling.\nGPT-3 labeling achieves better results in the low-cost regime, but has a gap with human labeling when enough money is spent on data collection. This implies the following inequation, although to what extent “a lot” or “noisy” means depends on the task details.\nA lot of high-quality data > A lot of noisy data > A little high quality data\n.\nGPT-3 labeling technique improves the classification performance in the low-cost regime. (Image source:\nWang et al. 2021\n)\nLanguage Model as Data Generator\n#\nIf enough training dataset for text classification tasks are available, we can fine-tune language models to synthesize more training samples conditioned on labels (\nAnaby-Tavor et al. 2019\n,\nKumar et al. 2021\n).\nLanguage-model-based data augmentation\n(\nLAMBADA\n;\nAnaby-Tavor et al. 2019\n) takes such an idea, where the process involves fine-tuning both a classifier and a sample generation model.\nTrain a baseline classifier using the existing training dataset: $h = \\mathcal{A}(\\mathcal{D}_\\text{train})$.\nIndependently of step 1, a LM $\\mathcal{M}$ is fine-tuned on $\\mathcal{D}_{\\text{train}}$ to obtain $\\mathcal{M}_{\\text{tuned}}$.\nSynthesize a labeled dataset $\\mathcal{D}^*$ by generating the continuation of the sequence\ny[SEP]\nuntil\nEOS\nusing $\\mathcal{M}_\\text{tuned}$.\nFilter synthesized dataset by,\n(1) Verifying that the predicted label is correct $h(x)=y$;\n(2) Selecting the top ranked samples when they are ranked by the classifier probability. $\\mathcal{D}_\\text{syn} \\subset \\mathcal{D}^*$. They generate 10x more samples needed for augmentation and only the top 10% synthesized samples with highest confidence scores remain.\nThe final classifier is trained on $\\mathcal{D}_\\text{syn} \\cup \\mathcal{D}_\\text{train}$ . The process can be repeated multiple times, but it is unclear whether the benefit would quickly diminish or the repetitive process would bring in self-bias.\nAccuracy of LAMBADA vs. other generative approaches over all datasets and classifiers. (Image source:\nAnaby-Tavor et al. 2019\n)\nTo simplify LAMBADA, we can actually remove the dependency of a fine-tuned generation model and an existing training dataset of a decent size (\nStep 2\nabove).\nUnsupervised data generation\n(\nUDG\n;\nWang et al. 2021\n) relies on few-shot prompting on a large pretrained language model to generate high-quality synthetic data for training. Opposite to the above approach where LM is asked to predict $y$ given $\\mathbf{x}$, UDG instead synthetizes the inputs $\\mathbf{x}$ given labels $y$. Then a task-specific model is trained on this synthetic dataset.\nSchick & Schutze (2021)\nproposed a similar idea but on the NLI task instead of classification, asking PLM to write sentence pairs that are similar or different while the model is prompted with task-specific instructions.\nIllustration of the unsupervised data generation (UDG) framework. (Image source:\nWang et al., 2021\n)\nThe few-shot prompts of UDG contain a small number of unlabeled examples, as well as a task-specific natural language description of the desired label. Because some generated examples are noisy, they implemented\nnoisy label annealing\n(\nNLA\n) techniques to filter potentially misaligned samples out during the training processes. NLA gradually removes noisy training signals in time during training when the model starts to disagree with its pseudo label with high confidence. At each training step $t$, a given example $(\\mathbf{x}_i, \\hat{y}_i)$ is considered noisy and should be removed if:\nThe model predicted probability is higher than a threshold $p(\\bar{y}_i \\vert \\mathbf{x}_i) > \\mu_t$ where $\\bar{y}_i = \\arg\\max_y p(y \\vert \\mathbf{x}_i)$;\nAnd the predicted label is different from the synthetic label, $\\bar{y}_i \\neq \\hat{y}_i$.\nNote that the threshold $\\mu_t$ is time-dependent, initialized as 0.9 and then gradually annealed to $1/\\text{num_of_classes}$ in time.\nAs shown in their experiments, the improvement of UDG over few-shot inference is quit significant, where NLA brings in some extra boost. The results are even comparable with supervised fine-tuning on several cases.\nComparison of accuracy of UDG and other methods on different classification datasets. (Image source:\nWang et al., 2021\n)\nHan et al (2021)\nachieved SOTA results on translation tasks using few-shot data generation, distillation and back-translation. The proposed method contains the following steps, assuming no access to paired translation data:\nZero-shot Generation.\nFirst use the zero-shot translation ability of a pre-trained LM to generate translations for a small set of unlabeled sentences.\nFew-shot Generation.\nThen amplify these zero-shot translations by using them as few-shot demonstrations to gather an even larger synthetic dataset.\nDistillation.\nFine-tune the model on this dataset. The translation task is formulated as a language modeling task\n[L1] [[TRANSLATE]] [L2] .\ngiven a pair of two sequences\n\nin two different languages. At test-time, the LM is prompted with\n[L1] [[TRANSLATE]] [L2]\nand a candidate translation\n\nis parsed from the sampled completion.\nBack-translation.\nContinue fine-tuning on the back-translation dataset where the order of samples is reversed,\n\n.\nStep 1-4 can be repeated.\nAlgorithm of using distillation and back-translation to train a language model on translation tasks. (Image source:\nHan et al. 2021\n)\nThe success of the above method depends on a good pretrained LM to kick off the initial translation dataset. Iterative few-shot generation and distillation with back-translation is an effective way to extract and refine the translation capability out of a pretrained LM and further to distill that into a new model.\nComparison of BLEU scores of the translation models of different training runs using: only distillation, back-translation, both and with more monolingual training data. (Image source:\nHan et al. 2021\n)\nHow to Quantify Generated Data Quality?\n#\nGiven all the generated data, either by data augmentation or data synthesis, how can we quantify data quality in terms of how they improve model generalization?\nGontijo-Lopes et al. (2020)\nintroduced two dimensions to track, affinity and diversity.\nAffinity\nis a model-sensitive metric for\ndistribution shift\n, quantifying how much an augmentation shifts the training data distribution from what a model learned.\nDefinition: The performance difference between the model tested on clean data vs augmented data, while the model is trained on clean data.\nAs a comparison, KL can also measure distribution shift but does not consider the model performance.\nDiversity\nis a measure of\naugmentation complexity\n, measuring the complexity of the augmented data with respect to the model and learning procedure.\nDefinition: The final training loss of a model trained with a given augmentation.\nAnother potential diversity measure is the entropy of the transformed data.\nA third potential diversity measure is the training time needed for a model to reach a given training accuracy threshold.\nAll three metrics above are correlated.\nThe final model performance is dependent on both metrics to be high enough.\n(a) Left: A scatter plot of affinity vs diversity metric, where each point represents a different augmentation method and its color indicates the final test accuracy. (b) Right: The conceptual illustration of the relationship between clean and augmented data in different regions of affinity and diversity metrics. (Image source:\nGontijo-Lopes et al. 2020\n)\nThere are many quantitative metrics on relevancy and diversity, in different formations depending on whether a reference is available, such as perplexity, BLEU for text and inception score for images. I’m skipping the list of concrete quantitative metrics on quality here, given it could be very long.\nTraining with Noisy Data\n#\nIt is convenient to collect a large amount of noisy data via model generation or data augmentation, but it is hard to guarantee that augmented and generated data can be 100% accurate. Knowing that deep neural networks can easily overfit noisy labels and “memotize” corrupted labels, we can apply the techniques for training on noisy labels (\nnoise-robust training\n) when using generated data to stabilize and optimize the performance. Please check this\nsurvey paper (Song et al. 2021)\non learning from noisy labels for a more thorough coverage of related work.\nRegularization and Robust Architecture\n#\nGenerally speaking, mechanisms designed for avoiding overfitting should help improve training robustness when working with moderately noisy data, such as weight decay, dropout, batch normalization. In fact, good data augmentation (i.e. only non-essential attributes are modified) can be considered as a way of regularization as well.\nA different approach is to enhance the network with a dedicated\nnoisy adaptation layer\nto approximate the unknown projection of label corruption (\nSukhbaatar et al. 2015\n,\nGoldberger & Ben-Reuven, 2017\n).\nSukhbaatar et al. (2015)\nintroduced an extra linear layer $Q$ into the network architecture to adapt the predictions to match the noisy label distribution. The noise matrix $Q$ is initially\nfixed\nto the identity function while only the base model parameters is updated. After some time, $Q$ starts to be updated and expected to capture the noise in the data. The noise matrix is trained with regularization to encourage it to match the noise distribution while keeping the base model prediction accurate for true labels.\n(a) Left: A noise matrix $Q$ is added between softmax and the final output for the loss. (b) Right: The noise matrix $Q$ is fixed at the identity function initially and only gets updated with regularization after some training. (Image source:\nSukhbaatar et al. 2015\n)\nHowever, it is hard to guarantee such a noise matrix layer would only capture the noise transition distribution and it is actually non-trivial to learn.\nGoldberger & Ben-Reuven (2017)\n) proposed to add an additional softmax layer end-to-end with the base model and apply the\nEM algorithm\nby treating the correct labels as latent random variable and the noise processes as a communication channel with unknown parameters.\nRobust Learning Objective\n#\nBesides the most commonly used cross entropy loss, some other choices of learning objectives are shown to be more robust to noisy labels.\nFor example,\nMAE\n(mean absolute error) is more robust to noisy labels than CCE (categorical cross entropy), as it treats every sample equally (\nGhosh et al. 2017\n). Lack of different weighting among training samples of MAE lead to significantly longer training time. Motivated by the tradeoff between MAE and CCE,\nZhang & Sabuncu (2018)\nproposed\ngeneralized cross entropy\n(\nGCE\n), a generalization of CCE loss to be robust to noisy data.\nTo exploit the benefits of both the noise-robustness provided by MAE and the implicit weighting scheme of CCE, GCE adopts the the negative Box-Cox transformation as a loss function:\n$$\n\\mathcal{L}_q(f(\\mathbf{x}_i, y_i = j)) = \\frac{1 - f^{(j)}(\\mathbf{x}_i)^q}{q}\n$$\nwhere $f^{(j)}$ denotes the $j$-th element of $f(.)$ and $q \\in (0, 1]$.\n$\\mathcal{L}_q$ is equivalent to CCE when $q \\to 0$ and becomes MAE when $q=1$. Empirical experiments show that there exists a threshold of $q$ with which overfitting never emerges and the noisier the data the higher such a threshold should be.\nGiven true and predicted labels, $y_i, \\hat{y}_i \\in \\{0, 1\\}$ and let $u_i=y_i \\cdot \\hat{y}_i$, the\nzero-one loss\n, $\\mathcal{L}_{01}(\\mathbf{u}) = \\sum_{i=1}^n \\mathbb{1}[u_i < 0]$, is another learning subjective shown to be robust to noisy data. Minimizing the empirical risk with the zero-one loss is shown to be equivalent to minimizing the empirical adversarial (worse-case) risk (\nHu et al 2018\n). Because the worst-case risk is the upper bound of the classification risk of the clean data distribution, minimizing the worst-case risk can lead to decreased true risk, which makes the zero-one loss especially robust. However, the zero-one loss is non-differentiable and cannot be optimized directly. One solution is to approximate an\nupper bound\nof the zero-one loss and to minimize the upper bound loss instead.\nThe\nhinge loss\n, $\\mathcal{L}_\\text{hinge}(\\mathbf{u}) = \\sum_{i=1}^n \\max(0, 1 - u_i)$, defines a rough upper bound of the zero-one loss.\nLyu & Tsang (2020)\nproposed a\ncurriculum loss\n(\nCL\n), which is a tighter upper bound compared to a conventional surrogate loss like the hinge loss, $\\mathcal{L}_\\text{01}(\\mathbf{u}) \\leq \\mathcal{L}_\\text{CL}(\\mathbf{u}) \\leq \\mathcal{L}_\\text{hinge}(\\mathbf{u})$.\n$$\n\\mathcal{L}_\\text{CL}(\\mathbf{u}) = \\min_{\\mathbf{w}\\in\\{0,1\\}^n}\\max(\\sum_{i=1}^n w_i \\ell(u_i), n - \\sum_{i=1}^n w_i + \\sum_{i=1}^n\\mathbb{1}[u_i < 0])\n$$\nwhere $\\ell(u_i)$ is a base surrogate loss for the zero-one loss (e.g. hinge loss) and the optimal weighting variable $\\mathbf{w}$ is to be learned.\nGiven a label corruption rate $\\rho$, the\nnoise pruned curriculum loss\n(\nNPCL\n) is constructed based on the intuition that an ideal model should correctly classify $n(1-\\rho)$ samples with clean labels but misclassify $n\\rho$ corrupted labels. If $\\rho$ is a known prior, we would know how many samples (with largest losses) to be pruned. Assuming $\\ell(u_1) \\leq \\dots \\leq \\ell(u_n)$, then $u_{n(1-\\rho)+1} = \\dots = u_n =0$ and the following NPCL is the basic CL for only $n(1-\\rho)$ samples:\n$$\n\\text{NPCL}(\\mathbf{u}) = \\min_{\\mathbf{w}\\in\\{0,1\\}^{n(1-\\rho)}} \\max(\\sum_{i=1}^{n(1-\\rho)} w_i \\ell(u_i), n(1-\\rho) - \\sum_{i=1}^{n(1-\\rho)} w_i)\n$$\nWhen experimenting on CIFAR-10, NPCL is comparable with GCE and performs better when the noise rate increases.\nLabel Correction\n#\nSince it is known some labels are incorrect, noise-robust training can explicitly take the label correction into consideration.\nOne approach is to rely on the estimation of a noise transition matrix and use that to correct the forward or backward loss, named\nF-correction\n(\nPatrini et al. 2017\n). Let’s first assume that there are $k$ classes and the noise transition matrix $C \\in [0, 1]^{k\\times k}$ is observable and the label flipping probability does not depend on the sample input but only the label (i.e. known as random classification noise, RCN). Let $\\tilde{y}$ denote a corrupted label. Each entry of $C$ represents the probability of one label flipping to another\n1\n,\n$$\nC_{ij} = p(\\tilde{y}= j \\vert y =i, \\mathbf{x}) \\approx p(\\tilde{y}= j \\vert y =i)\n$$\nThen we can proceed a forward label correction procedure to incorporate the prior knowledge of noisy transition matrix into the prediction.\n$$\n\\begin{aligned}\n\\mathcal{L}(\\hat{p}(\\tilde{y}\\vert\\mathbf{x}), y)\n&= - \\log \\hat{p}(\\tilde{y}=i\\vert\\mathbf{x}) \\\\\n&= - \\log \\sum_{j=1}^k p(\\tilde{y}=i\\vert y=j) \\hat{p}(y=j\\vert\\mathbf{x}) \\\\\n&= - \\log \\sum_{j=1}^k C_{ji} \\hat{p}(y=j\\vert\\mathbf{x})\n\\end{aligned}\n$$\nIn matrix form, we have $\\mathcal{L}(\\hat{p}(y \\vert \\mathbf{x})) = - \\log C^\\top \\hat{p}(y \\vert \\mathbf{x})$. However, such a noise transition matrix is usually\nunknown\n. If we have access to a clean dataset, the noise matrix $C$ can be estimated (\nHendrycks et al. 2018\n) by calculating confusion matrix on the clean data. Let’s denote a clean trusted dataset as $\\mathcal{D}_c$ and a noisy dataset as $\\mathcal{D}_n$ going forward.\n$$\n\\hat{C}_{ij}\n= \\frac{1}{\\vert \\mathcal{A}_i\\vert} \\sum_{\\mathbf{x} \\in \\mathcal{A}_i} \\hat{p}(\\tilde{y}=j \\vert y=i, \\mathbf{x})\n\\approx p(\\tilde{y}=j \\vert y=i)\n$$\nwhere $\\mathcal{A}_i$ is a subset of data points from $\\mathcal{D}_c$ with label $i$.\nLet $f(x) = \\hat{p}(\\tilde{y} \\vert \\mathbf{x}; \\theta)$ and this model should be trained with $\\mathcal{L}(f(\\mathbf{x}), y)$ on clean data $\\mathcal{D}_c$ and with $\\mathcal{L}(\\hat{C}^\\top f(\\mathbf{x}), \\hat{y})$ on noisy data $\\mathcal{D}_n$.\nAlgorithm of gold loss correction (GLC), estimating the noise transition matrix with a trusted dataset. (Image source:\nHendrycks et al. 2018\n)\nIf the trusted training dataset $\\mathcal{D}_c$ gets large, we can train a neural network only on clean data and\ndistill\nits knowledge into the primary model (i.e. the final model to make predictions at test time) using corrected\npseudo labels\n(\nLi et al. 2017\n). The primary model is trained on the entire dataset, $\\mathcal{D} = \\mathcal{D}_c \\cup \\mathcal{D}_n$. Optionally the “side” information of label relations in the knowledge graph, if available, can be incorporated into distillation to help the robustness of the predictions of the network that is trained on limited data.\nThe label correction distillation works as following:\nFirst train an auxiliary model $f_c$ from the small clean dataset $\\mathcal{D}_c$ to provide a soft label for each sample $x_i$, $s_i = \\delta(f_c(\\mathbf{x}_i)/T)$ is the sigmoid activation with temperature $T$.\nBecause the clean dataset is not large, $f_c$ is likely to overfit,\nLi et al. (2017)\nturn to a knowledge graph $\\mathcal{G}$ that defines the relations in the label space and\npropagate\nthe prediction among labels accordingly. The new soft label is donated as $\\hat{s}_i = \\mathcal{G}(s_i)$.\nThe primary model $f$ is trained with predictions from $f_c$ to imitate,\n$$\n\\mathcal{L}(y_i, f(\\mathbf{x}_i)) = \\text{CE}(\\underbrace{\\lambda y_i + (1 - \\lambda) \\hat{s}_i}_\\text{pseudo label}, f(\\mathbf{x}_i))\n$$\nSample Reweighting and Selection\n#\nSome samples may be more likely to have inaccurate labels than others. Such estimation gives us intuition on which samples should be weighted less or more in the loss function. However, considering two types of biases in training data, class imbalance and noisy labels, there is actually a contradictory preference — We would prefer samples with larger loss to balance the label distribution but those with smaller loss for mitigating the potential noise. Some work (\nRen et al. 2018\n) thus argue that in order to learn general forms of training data biases, it is\nnecessary\nto have\na small unbiased validation\nto guide training. The sample reweighting methods presented in this section all assume access to a small trusted set of clean data.\nConsidering a binary classification task with random classification noise, $y, \\hat{y} \\in \\{-1, +1\\}$, the label flipping probabilities, $\\rho_{-1}, \\rho_{+1} \\in [0, 0.5)$, are defined as:\n$$\n\\rho_{-1} = P(\\tilde{y} = +1 \\vert y=-1)\\quad\\rho_{+1} = P(\\tilde{y}=-1 \\vert y =+1)\n$$\nLiu & Tao (2015)\napplies\nimportance reweighting\nto adjust the weighted distribution of observed $\\hat{y}$ to match the distribution of unobservable $y$. Let $\\mathcal{D}$ be the true data distribution and $\\mathcal{D}_\\rho$ be the corrupted version.\n$$\n\\begin{aligned}\n\\mathcal{L}_{\\ell,\\mathcal{D}}(f)\n&= \\mathbb{E}_{(\\mathbf{x},y)\\sim \\mathcal{D}}[\\ell(f(\\mathbf{x}), y)] \\\\\n&= \\mathbb{E}_{(\\mathbf{x},\\tilde{y})\\sim \\mathcal{D}_\\rho} \\Big[ \\frac{P_\\mathcal{D}(\\mathbf{x}, y=\\tilde{y})}{P_{\\mathcal{D}_\\rho}(\\mathbf{x}, \\tilde{y})} \\ell(f(\\mathbf{x}), \\tilde{y}) \\Big] \\\\\n&= \\mathbb{E}_{(\\mathbf{x},\\tilde{y})\\sim \\mathcal{D}_\\rho} \\Big[ \\frac{P_\\mathcal{D}(y=\\tilde{y} \\vert \\mathbf{x})}{P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x})} \\ell(f(\\mathbf{x}), \\tilde{y}) \\Big] & \\text{; because }P_\\mathcal{D}(\\mathbf{x})=P_{\\mathcal{D}_\\rho}(\\mathbf{x}) \\\\\n&= \\mathbb{E}_{(\\mathbf{x},\\tilde{y})\\sim \\mathcal{D}_\\rho} [ w(\\mathbf{x}, \\hat{y})\\ell(f(\\mathbf{x}), \\tilde{y}) ]\n= \\mathcal{L}_{w\\ell,\\mathcal{D}}(f)\n\\end{aligned}\n$$\nBecause,\n$$\n\\begin{aligned}\nP_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x})\n&= P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x}) P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert y=\\tilde{y}) +\nP_\\mathcal{D}(y = - \\tilde{y} \\vert \\mathbf{x}) P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert y = - \\tilde{y}) \\\\\n&= P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x}) (1 - P_{\\mathcal{D}_\\rho}(- \\tilde{y} \\vert y=\\tilde{y})) +\n(1 - P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x})) P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert y = - \\tilde{y}) \\\\\n&= P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x}) (1 - \\rho_{\\tilde{y}}) +\n(1 - P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x})) \\rho_{-\\tilde{y}} \\\\\n&= P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x})(1 - \\rho_{\\tilde{y}} - \\rho_{-\\tilde{y}}) + \\rho_{-\\tilde{y}}\n\\end{aligned}\n$$\nThus the weight assigned to a noisy sample is,\n$$\nw(x, \\tilde{y})\n= \\frac{P_\\mathcal{D}(y=\\tilde{y} \\vert \\mathbf{x})}{P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x})}\n= \\frac{P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x}) - \\rho_{-\\tilde{y}}}{(1-\\rho_0-\\rho_1) P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x})}\n$$\nwhere $P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x})$ can be estimated using a simple logistic regression, but estimating the note rates is more challenging. Naive cross-validation can work out but is costly as the quality depends on the amount of trusted labels available. The paper approximates the upper bounds for noise rates first, $\\rho_\\tilde{y} \\leq P_{\\mathcal{D}_\\rho}(- \\tilde{y} \\vert \\mathbf{x})$ and then use a mild assumption to efficiently estimate them, $\\hat{\\rho}_{\\tilde{y}} = \\min_{\\mathbf{x} \\in {\\mathbf{x}_1, \\dots, \\mathbf{x}_n}} \\hat{P}_{\\mathcal{D}_\\rho}(- \\tilde{y} \\vert \\mathbf{x})$. In their experiments, the advantage of importance reweighting only varies across datasets and is more beneficial when the noise rates are high in general.\nSample reweighting schemes can be learned by a separate network.\nLearning to reweight\n(\nL2R\n;\nRen et al. 2018\n) is a meta-learning approach to directly optimize the weights in pursuit of best validation performance on a known set of clean data. Each example gets assigned with the weight based on its gradient direction. The weighted loss to minimize $\\theta^*(\\mathbf{w})$ involves a set of training weights $\\{w_i\\}_{i=1}^n$ as unknown hyperparameters. These sample training weights $w_i$ are learned to minimize the loss on this unbiased validate set, $\\mathcal{D}_c = \\{x^\\text{valid}_j\\}_{j=1}^m$.\n$$\n\\begin{aligned}\n\\theta^{*}(\\mathbf{w}) &= \\arg\\min_\\theta \\sum_{i=1}^n w_i f(x_i; \\theta) \\\\\n\\text{where optimal }\\mathbf{w}^{*} &= \\arg\\min_{\\mathbf{w}, \\mathbf{w} \\geq \\mathbf{0}} \\frac{1}{m} \\sum_{j=1}^m f(\\mathbf{x}^\\text{valid}_j; \\theta^{*}(\\mathbf{w}))\n\\end{aligned}\n$$\nThe learning process involves two nested loops of optimization, so pretty expensive, 3x training time.\nIllustration of updates implemented by second order\nautomatic differentiation\n. (Image source:\nRen et al. 2018\n)\nThey ran experiments on (1) two-class MNIST to test the robustness of L2R when the class distribution is imbalanced and (2) CIFAR-10 with noisy labels. L2R is shown to be better than other baseline methods at the time on both tasks.\nLeft: Imbalanced classes on MNIST (class 4 and 9); Right: Effect of the number of clean samples. Task is on CIFAR-10 with 40% of data flipped to label 3. (Image source:\nRen et al. 2018\n)\nMentorNet\n(\nJiang et al. 2018\n) uses teach-student curriculum learning to weight data. It incorporates two different networks, a mentor and a student. The mentor network provides a data-driven curriculum (i.e. sample training weighting scheme) for the student to focus on learning likely correct labels.\nLet $g_\\psi$ be the MentorNet parameterized by $\\psi$ , $f_\\theta$ be the StudentNet parametrized by $\\theta$ and $G$ be a predefined curriculum parameterized by $\\lambda$. Given the training data $\\mathcal{D} = \\{(\\mathbf{x}_i, y_i)\\}_{i=1}^n$ for a $k$-class classification task, the MentorNet needs to predict a time-varying latent weight variable $\\mathbf{w} \\in [0, 1]^{n \\times k}$ to guide the learning of StudentNet, taking an intermediate feature processed by StudentNet $f$ , $\\mathbf{z}_i = \\phi_{f_\\theta}(\\mathbf{x}_i, y_i)$:\n$$\ng_{\\psi^{*}}(\\mathbf{z}_i) = \\arg\\min_{w_i \\in [0,1]} \\mathcal{L}(\\theta, \\mathbf{w}), \\forall i \\in [1, n]\n$$\nStudentNet learns to minimize the following learning objective,\n$$\n\\begin{aligned}\n\\mathcal{L}(\\theta, \\mathbf{w})\n&= \\frac{1}{n}\\sum_{i=1}^n \\mathbf{w}_i^\\top \\ell(y_i, f_\\theta(\\mathbf{x}_i)) + G_\\lambda(\\mathbf{w}) + \\alpha |\\theta|^2_2 \\\\\n&= \\frac{1}{n}\\sum_{i=1}^n g_\\psi(\\mathbf{z}_i)^\\top \\ell_i + G_\\lambda(\\mathbf{w}) + \\alpha |\\theta|^2_2 & \\text{; Let }\\ell_i = \\ell(y_i, f_\\theta(\\mathbf{x}_i)) \\\\\n\\end{aligned}\n$$\nThe mentor network $g_\\psi$ is trained with cross entropy on the input $(\\phi_{f_\\theta}(\\mathbf{x}_i, y_i), w^{*}_i)$ , where $v^*_i=1$ if $y_i$ is known to be a correct label, otherwise 0. The architecture of MentorNet does not have to be very complicated. In the paper, they adopted a LSTM layer to capture the prediction variance in time.\nModel architecture of MentorNet and StudentNet which are trained simultaneously, where MentorNet predicts the sample weights for StudentNet to train on. (Image source:\nJiang et al. 2018\n)\nDifferent from MentorNet where one network explicitly learns weighting scheme and curriculum for the other network,\nCo-teaching\n(\nHan et al. 2018\n) trains two neural networks, $f_1$ and $f_2$, simultaneously and lets them teach each other by feeding data to each other selectively. Co-teaching consists of three steps:\nFirst, each network feeds forward the current mini-batch and selects samples with potentially clean labels;\nThen two networks exchange information on which samples in the batch should be used for training. Small-loss instances are selected as they are more likely to be associated with correct labels. The percentage of the batch to select is determined by a time-dependent function $R(T)$. The value of $R(T)$ decreases in time because the network is more likely to overfit and memorize noisy labels as training progresses and thus we use a smaller sampling percentage to keep the selected data quality high.\nFinally, each network runs back-propagation updates with the data selected by its peer.\nAccording to their experiments, co-teaching performs better than\nF-correction\nwhere the noise rates are high or the corruption transition matrix is not symmetric.\nAlgorithm of co-teaching in which two networks are trained separately in parallel and each selects samples for the other to train on. (Image source:\nHan et al. 2018\n)\nCitation\n#\nCited as:\nWeng, Lilian. (Apr 2022). Learning with not enough data part 3: data generation. Lil’Log. https://lilianweng.github.io/posts/2022-04-15-data-gen/.\nOr\n@article{weng2022datagen,\n title = \"Learning with not Enough Data Part 3: Data Generation\",\n author = \"Weng, Lilian\",\n journal = \"Lil'Log\",\n year = \"2022\",\n month = \"Apr\",\n url = \"https://lilianweng.github.io/posts/2022-04-15-data-gen/\"\n}\nReference\n#\n[1] Zhang et al.\n“Adversarial AutoAgument”\nICLR 2020.\n[2] Kumar et al.\n“Data Augmentation using Pre-trained Transformer Models.”\nAACL 2020 Workshop.\n[3] Anaby-Tavor et al.\n“Not enough data? Deep learning to rescue!”\nAAAI 2020.\n[4] Wang et al.\n“Want To Reduce Labeling Cost? GPT-3 Can Help.”\nEMNLP 2021.\n[5] Wang et al.\n“Towards Zero-Label Language Learning.”\narXiv preprint arXiv:2109.09193 (2021).\n[6] Schick & Schutze.\nGenerating Datasets with Pretrained Language Models.\"\nEMNLP 2021.\n[7] Han et al.\n“Unsupervised Neural Machine Translation with Generative Language Models Only.”\narXiv preprint arXiv:2110.05448 (2021).\n[8] Guo et al.\n“Augmenting data with mixup for sentence classification: An empirical study.”\narXiv preprint arXiv:1905.08941 (2019).\n[9] Ekin D. Cubuk et al.\n“AutoAugment: Learning augmentation policies from data.”\narXiv preprint arXiv:1805.09501 (2018).\n[10] Daniel Ho et al.\n“Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules.”\nICML 2019.\n[11] Cubuk & Zoph et al.\n“RandAugment: Practical automated data augmentation with a reduced search space.”\narXiv preprint arXiv:1909.13719 (2019).\n[12] Zhang et al.\n“mixup: Beyond Empirical Risk Minimization.”\nICLR 2017.\n[13] Yun et al.\n“CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features.”\nICCV 2019.\n[14] Kalantidis et al.\n“Mixing of Contrastive Hard Negatives”\nNeuriPS 2020.\n[15] Wei & Zou.\n“EDA: Easy data augmentation techniques for boosting performance on text classification tasks.”\nEMNLP-IJCNLP 2019.\n[16] Kobayashi.\n“Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations.”\nNAACL 2018\n[17] Fang et al.\n“CERT: Contrastive self-supervised learning for language understanding.”\narXiv preprint arXiv:2005.12766 (2020).\n[18] Gao et al.\n“SimCSE: Simple Contrastive Learning of Sentence Embeddings.”\narXiv preprint arXiv:2104.08821 (2020). [\ncode\n]\n[19] Shen et al.\n“A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation.”\narXiv preprint arXiv:2009.13818 (2020) [\ncode\n]\n[20] Wang & van den Oord.\n“Multi-Format Contrastive Learning of Audio Representations.”\nNeuriPS Workshop 2020.\n[21] Wu et al.\n“Conditional BERT Contextual Augmentation”\narXiv preprint arXiv:1812.06705 (2018).\n[22 Zhu et al.\n“FreeLB: Enhanced Adversarial Training for Natural Language Understanding.”\nICLR 2020.\n[23] Affinity and Diversity: Quantifying Mechanisms of Data Augmentation\nGontijo-Lopes et al. 2020 (\nhttps://arxiv.org/abs/2002.08973\n)\n[24] Song et al.\n“Learning from Noisy Labels with Deep Neural Networks: A Survey.”\nTNNLS 2020.\n[25] Zhang & Sabuncu.\n“Generalized cross entropy loss for training deep neural networks with noisy labels.”\nNeuriPS 2018.\n[26] Goldberger & Ben-Reuven.\n“Training deep neural-networks using a noise adaptation layer.”\nICLR 2017.\n[27] Sukhbaatar et al.\n“Training convolutional networks with noisy labels.”\nICLR Workshop 2015.\n[28] Patrini et al.\n“Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach”\nCVPR 2017.\n[29] Hendrycks et al.\n“Using trusted data to train deep networks on labels corrupted by severe noise.”\nNeuriPS 2018.\n[30] Zhang & Sabuncu.\n“Generalized cross entropy loss for training deep neural networks with noisy labels.”\nNeuriPS 2018.\n[31] Lyu & Tsang.\n“Curriculum loss: Robust learning and generalization against label corruption.”\nICLR 2020.\n[32] Han et al.\n“Co-teaching: Robust training of deep neural networks with extremely noisy labels.”\nNeuriPS 2018. (\ncode\n)\n[33] Ren et al.\n“Learning to reweight examples for robust deep learning.”\nICML 2018.\n[34] Jiang et al.\n“MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels.”\nICML 2018.\n[35] Li et al.\n“Learning from noisy labels with distillation.”\nICCV 2017.\n[36] Liu & Tao.\n“Classification with noisy labels by importance reweighting.”\nTPAMI 2015.\n[37] Ghosh, et al.\n“Robust loss functions under label noise for deep neural networks.”\nAAAI 2017.\n[38] Hu et al.\n“Does Distributionally Robust Supervised Learning Give Robust Classifiers? “\nICML 2018.\n$y=i$ is not a technically correct way to annotate a label being a certain value, since we usually use one-hot encoding (i.e. $\\mathbf{y} = \\mathbf{e}_i$). We use this form for simplicity.\n↩︎", "metadata": { "title": "Learning with not Enough Data Part 3: Data Generation | Lil'Log", "description": "Here comes the Part 3 on learning with not enough data (Previous: Part 1 and Part 2). Let’s consider two approaches for generating synthetic data for training.\n\nAugmented data. Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a previous post on contrastive learning. For the sake of post completeness, I duplicate the section on data augmentation here with some edits.\nNew data. Given few or even no data points, we can rely on powerful pretrained models to generate a number of new data points. This is especially true in recent years given the fast progress in large pretrained language models (LM). Few shot prompting is shown to be effective for LM to learn within context without extra training.\n\nData Augmentation\nThe goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.", "keywords": "data, data-generation, data-augmentation, data-quality", "author": "Lilian Weng", "og:title": "Learning with not Enough Data Part 3: Data Generation", "og:description": "Here comes the Part 3 on learning with not enough data (Previous: Part 1 and Part 2). Let’s consider two approaches for generating synthetic data for training.\n\nAugmented data. Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a previous post on contrastive learning. For the sake of post completeness, I duplicate the section on data augmentation here with some edits.\nNew data. Given few or even no data points, we can rely on powerful pretrained models to generate a number of new data points. This is especially true in recent years given the fast progress in large pretrained language models (LM). Few shot prompting is shown to be effective for LM to learn within context without extra training.\n\nData Augmentation\nThe goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.", "og:type": "article", "og:url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/", "twitter:card": "summary", "twitter:title": "Learning with not Enough Data Part 3: Data Generation", "twitter:description": "Here comes the Part 3 on learning with not enough data (Previous: Part 1 and Part 2). Let’s consider two approaches for generating synthetic data for training.\n\nAugmented data. Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a previous post on contrastive learning. For the sake of post completeness, I duplicate the section on data augmentation here with some edits.\nNew data. Given few or even no data points, we can rely on powerful pretrained models to generate a number of new data points. This is especially true in recent years given the fast progress in large pretrained language models (LM). Few shot prompting is shown to be effective for LM to learn within context without extra training.\n\nData Augmentation\nThe goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.", "article:section": "posts", "article:published_time": "2022-04-15T15:10:30-07:00", "article:modified_time": "2022-04-15T15:10:30-07:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/1cc4df7eb2c7/code/body-000.txt", "chars": 257, "language": "text" } ], "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/EDA-exp1.png", "bytes": 299081, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-b1e26be047c763592a3a.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/EDA-exp1.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GPT3-data-gen.png", "bytes": 178022, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-364dc68a08807b5baf8a.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GPT3-data-gen.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GPT3-data-gen-exp.png", "bytes": 979053, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-2a8613e8ff2f07c9bbde.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GPT3-data-gen-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/LAMBADA.png", "bytes": 163034, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-1e49c821edae4f24f95b.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/LAMBADA.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/UDG.png", "bytes": 396874, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-773377f93dad7a113410.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/UDG.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/UDG-exp.png", "bytes": 103983, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-82e84409bcacc2fc600e.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/UDG-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/back-translation-data-gen.png", "bytes": 161004, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-3800c7c1476957d08f84.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/back-translation-data-gen.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/back-translation-data-gen-exp.png", "bytes": 107466, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-9ca541330b72888db6f9.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/back-translation-data-gen-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/affinity-diversity.png", "bytes": 257673, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-6945c914870c32780b47.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/affinity-diversity.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/noise-adaptation-layer.png", "bytes": 193590, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-1e20893406355a07939d.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/noise-adaptation-layer.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GLC.png", "bytes": 283685, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-6185b7cd92536235d8de.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GLC.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/L2R-backprop.png", "bytes": 241069, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-67e6e4a22214ef58e33c.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/L2R-backprop.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/L2R-exp.png", "bytes": 812915, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-7d52e35a3da415faf3df.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/L2R-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/MentorNet.png", "bytes": 260104, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-e2db6011ba9c735f3718.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/MentorNet.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/co-teaching.png", "bytes": 313297, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-cb3814a9fb4722b3289a.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/co-teaching.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/", "label": "Part 1", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2022-02-20-active-learning/", "label": "Part 2", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2021-05-31-contrastive/", "label": "previous post", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2019-01-31-lm/", "label": "language models (LM)", "type": "a" }, { "href": "https://pillow.readthedocs.io/en/stable/reference/ImageOps.html", "label": "PIL.ImageOps", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2019-05-05-domain-randomization/", "label": "AutoAugment", "type": "a" }, { "href": "https://arxiv.org/abs/1805.09501", "label": "Cubuk, et al. 2018", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2020-08-06-nas/", "label": "neural architecture search", "type": "a" }, { "href": "https://arxiv.org/abs/1912.11188", "label": "Zhang, et al 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1909.13719", "label": "Cubuk et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1905.05393", "label": "Ho et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1711.09846", "label": "Jaderberg et al, 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1904.12848", "label": "Xie et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1710.09412", "label": "Zhang et al., 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1905.04899", "label": "Yun et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1708.04552", "label": "DeVries & Taylor 2017", "type": "a" }, { "href": "https://arxiv.org/abs/2010.01028", "label": "Kalantidis et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1901.11196", "label": "Wei & Zou 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1805.06201", "label": "Kobayashi, 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1812.06705", "label": "Xing Wu et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1905.08941", "label": "Guo et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1909.11764", "label": "Zhu et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/2103.06508", "label": "Wang & van den Oord (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/2104.08821", "label": "Guo et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2009.13818", "label": "Shen et al. (2020)", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2017-08-20-gan/", "label": "GAN", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2018-08-12-vae/", "label": "VAE", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2018-10-13-flow-models/", "label": "flow", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/", "label": "diffusion", "type": "a" }, { "href": "https://arxiv.org/abs/2108.13487", "label": "Wang et al. (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/1911.03118", "label": "Anaby-Tavor et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/2003.02245", "label": "Kumar et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2109.09193", "label": "Wang et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2104.07540", "label": "Schick & Schutze (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/2110.05448", "label": "Han et al (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/2002.08973", "label": "Gontijo-Lopes et al. (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/2007.08199", "label": "survey paper (Song et al. 2021)", "type": "a" }, { "href": "https://arxiv.org/abs/1406.2080", "label": "Sukhbaatar et al. 2015", "type": "a" }, { "href": "https://openreview.net/forum?id=H12GRgcxg", "label": "Goldberger & Ben-Reuven, 2017", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm", "label": "EM algorithm", "type": "a" }, { "href": "https://arxiv.org/abs/1712.09482", "label": "Ghosh et al. 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1805.07836", "label": "Zhang & Sabuncu (2018)", "type": "a" }, { "href": "https://arxiv.org/abs/1611.02041", "label": "Hu et al 2018", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Hinge_loss", "label": "hinge loss", "type": "a" }, { "href": "https://arxiv.org/abs/1905.10045", "label": "Lyu & Tsang (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/1609.03683", "label": "Patrini et al. 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1802.05300", "label": "Hendrycks et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1703.02391", "label": "Li et al. 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1803.09050", "label": "Ren et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1411.7718", "label": "Liu & Tao (2015)", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Automatic_differentiation", "label": "automatic differentiation", "type": "a" }, { "href": "https://arxiv.org/abs/1712.05055", "label": "Jiang et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1804.06872", "label": "Han et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/2005.12766", "label": "“CERT: Contrastive self-supervised learning for language understanding.”", "type": "a" }, { "href": "https://github.com/princeton-nlp/SimCSE", "label": "code", "type": "a" }, { "href": "https://github.com/dinghanshen/cutoff", "label": "code", "type": "a" }, { "href": "https://github.com/bhanML/Co-teaching", "label": "code", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:51:25Z", "source_url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/1805.09501", "bytes": 1138440, "reused": false, "local_path": "papers/repair/9a0542ec64439d672cc9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1912.11188", "bytes": 759445, "reused": false, "local_path": "papers/repair/4f4d45c6d87c67d3af9a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.13719", "bytes": 1263406, "reused": true, "local_path": "papers/repair/c267ff367c4770ddb6c0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.05393", "bytes": 873126, "reused": false, "local_path": "papers/repair/4634b6280b855d570ac8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1711.09846", "bytes": 2370925, "reused": false, "local_path": "papers/repair/3bbdbf9c8931b9ea9ea0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1904.12848", "bytes": 1625724, "reused": true, "local_path": "papers/repair/17af125202b45606d9ad.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1710.09412", "bytes": 740562, "reused": true, "local_path": "papers/repair/7af2d52fbf38a69ec251.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.04899", "bytes": 1682836, "reused": false, "local_path": "papers/repair/fec62c4fe8efdeebd877.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1708.04552", "bytes": 741919, "reused": false, "local_path": "papers/repair/c3d681ed19bf88f43a98.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2010.01028", "bytes": 970998, "reused": false, "local_path": "papers/repair/d4d3c2412459021d347a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1901.11196", "bytes": 365049, "reused": true, "local_path": "papers/repair/389e34d78a0045e6d861.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1805.06201", "bytes": 722323, "reused": false, "local_path": "papers/repair/aebd3f77254c72e012a3.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1812.06705", "bytes": 431373, "reused": false, "local_path": "papers/repair/3e737149c9ea8568055b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.08941", "bytes": 345640, "reused": false, "local_path": "papers/repair/646a76cd251b8253359a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.11764", "bytes": 257462, "reused": false, "local_path": "papers/repair/f5ea669209a03760ab22.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2103.06508", "bytes": 601155, "reused": false, "local_path": "papers/repair/a3681dc71a79ffecd499.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2104.08821", "bytes": 1672580, "reused": false, "local_path": "papers/repair/e7a40652bdd04cd4b25f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2009.13818", "bytes": 398629, "reused": false, "local_path": "papers/repair/40a2d9b786e78e3cf30e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2108.13487", "bytes": 5884494, "reused": false, "local_path": "papers/repair/13a1e2b1cf2e73968c6b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1911.03118", "bytes": 372575, "reused": false, "local_path": "papers/repair/e8abfe05a235151d6c3e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2003.02245", "bytes": 250909, "reused": false, "local_path": "papers/repair/ba7634f079f15d2367dd.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2109.09193", "bytes": 359282, "reused": false, "local_path": "papers/repair/5b433ceacb396f0baf99.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2104.07540", "bytes": 402533, "reused": false, "local_path": "papers/repair/ffa07cbf93071d00f1ed.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2110.05448", "bytes": 386252, "reused": false, "local_path": "papers/repair/62c1d70878370a3a749f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.08973", "bytes": 1161444, "reused": false, "local_path": "papers/repair/cfe2f6f56fb90471ae4d.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2007.08199", "bytes": 1299522, "reused": false, "local_path": "papers/repair/e31349c322026bf09dbb.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1406.2080", "bytes": 1181382, "reused": false, "local_path": "papers/repair/bd04e7310e34c7124557.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1712.09482", "bytes": 1146169, "reused": false, "local_path": "papers/repair/c7b83a3603919eb61881.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1805.07836", "bytes": 569770, "reused": false, "local_path": "papers/repair/1a42ffbbf4e704947ae3.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1611.02041", "bytes": 687019, "reused": false, "local_path": "papers/repair/ae376c87ceb11d4ea40f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.10045", "bytes": 2142637, "reused": false, "local_path": "papers/repair/75e3f3f69d9794770568.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1609.03683", "bytes": 763555, "reused": false, "local_path": "papers/repair/e8309ba701b2977efaca.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1802.05300", "bytes": 1049334, "reused": false, "local_path": "papers/repair/9237191e681f5b67f6cc.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1703.02391", "bytes": 446285, "reused": false, "local_path": "papers/repair/e7f30c1dcfc80c923753.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1803.09050", "bytes": 3044218, "reused": false, "local_path": "papers/repair/40a7062040ed9fa7132b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1411.7718", "bytes": 634569, "reused": false, "local_path": "papers/repair/d8403b69812a8c4b8935.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1712.05055", "bytes": 827072, "reused": false, "local_path": "papers/repair/9788972c8ad31eb5bbb5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1804.06872", "bytes": 1984230, "reused": false, "local_path": "papers/repair/ce399494158e733915af.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2005.12766", "bytes": 412104, "reused": true, "local_path": "papers/repair/bae8e6b74f13390eb0a6.pdf" } ], "actual_video_files": [], "body_chars": 40655, "expected_body_images": 15, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/1cc4df7eb2c7/pages/source.html", "local_html": "attachments/1cc4df7eb2c7/pages/article.html", "body_base_path": "attachments/1cc4df7eb2c7/pages/", "body_markdown_path": "attachments/1cc4df7eb2c7/pages/article.md" } }, { "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/", "status": "ok", "article_id": "76583c4985bd", "local_path": "series-follow/1602/repair/74bca82addbb696cd397.html", "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/PI-model.png", "bytes": 77905, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/22f8744911da963ff507.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/temperal-ensembling.png", "bytes": 83063, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/2431b04e51e95d44cbfc.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/mean-teacher.png", "bytes": 147122, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/3704ca968b8b036dd979.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/mean-teacher-results.png", "bytes": 449224, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/264593afb504dba40862.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/consistency-training-with-noisy-samples.png", "bytes": 270328, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/fe1f57c36a0c6b35ff30.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/ICT.png", "bytes": 299310, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/686784e7314725fbdcc3.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/UDA-image-results.png", "bytes": 375554, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/abc3c1e68e088b944f70.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/UDA-language-results.png", "bytes": 278212, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/20f5ad7f538bcfcaa51f.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/pseudo-label-segregation.png", "bytes": 3492681, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/09be84ee222f7edd8341.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/label-propagation.png", "bytes": 340862, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/41918aaaac4a38312f21.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/MPL-results.png", "bytes": 373018, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/151ce877219f628d821e.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/MixMatch.png", "bytes": 130110, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/52dcd8696c072fec5989.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/ReMixMatch.png", "bytes": 519536, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/60ae1da08ae38b4e5477.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/DivideMix.png", "bytes": 124124, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/1f3e79d5973995850e17.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/DivideMix-algo.png", "bytes": 548196, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/b8bdf2d20d1fcc25c810.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/FixMatch.png", "bytes": 218342, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/015dfefe2091cea2d5d5.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/FixMatch-results.png", "bytes": 306334, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/de572b70c4267a43c66a.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/self-training-pre-training.png", "bytes": 470728, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/cf42d45d1104e492f130.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/big-self-supervised-model.png", "bytes": 176722, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/452ce3c3d7e9a8712c06.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/big-self-supervised-model-results.png", "bytes": 411449, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/9d2b0d8db4abe7b366db.png" } ], "article": { "source_file": "0930", "source_platform": "个人博客", "author": "Lilian Weng / Lil'Log", "title_listed": "Learning with not Enough Data Part 1: Semi-Supervised Learning", "keywords": "半监督学习,少数据,标注", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/", "publish_date": "2021-12", "corpus_stage": "数据处理", "notes": "半监督学习系列综述之一", "id": "76583c4985bd", "collected_at": "2026-10-02T14:13:01", "title": "Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Log", "body_markdown": "When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.\nPre-training + fine-tuning\n: Pre-train a powerful task-agnostic model on a large unsupervised data corpus, e.g.\npre-training LMs\non free text, or pre-training vision models on unlabelled images via\nself-supervised learning\n, and then fine-tune it on the downstream task with a small set of labeled samples.\nSemi-supervised learning\n: Learn from the labelled and unlabeled samples together. A lot of research has happened on vision tasks within this approach.\nActive learning\n: Labeling is expensive, but we still want to collect more given a cost budget. Active learning learns to select most valuable unlabeled samples to be collected next and helps us act smartly with a limited budget.\nPre-training + dataset auto-generation\n: Given a capable pre-trained model, we can utilize it to auto-generate a lot more labeled samples. This has been especially popular within the language domain driven by the success of few-shot learning.\nI plan to write a series of posts on the topic of “Learning with not enough data”. Part 1 is on\nSemi-Supervised Learning\n.\nWhat is semi-supervised learning?\n#\nSemi-supervised learning uses both labeled and unlabeled data to train a model.\nInterestingly most existing literature on semi-supervised learning focuses on vision tasks. And instead pre-training + fine-tuning is a more common paradigm for language tasks.\nAll the methods introduced in this post have a loss combining two parts: $\\mathcal{L} = \\mathcal{L}_s + \\mu(t) \\mathcal{L}_u$. The supervised loss $\\mathcal{L}_s$ is easy to get given all the labeled examples. We will focus on how the unsupervised loss $\\mathcal{L}_u$ is designed. A common choice of the weighting term $\\mu(t)$ is a ramp function increasing the importance of $\\mathcal{L}_u$ in time, where $t$ is the training step.\nDisclaimer\n: The post is not gonna cover semi-supervised methods with focus on model architecture modification. Check\nthis survey\nfor how to use generative models and graph-based methods in semi-supervised learning.\nNotations\n#\nSymbol\nMeaning\n$L$\nNumber of unique labels.\n$(\\mathbf{x}^l, y) \\sim \\mathcal{X}, y \\in \\{0, 1\\}^L$\nLabeled dataset. $y$ is a one-hot representation of the true label.\n$\\mathbf{u} \\sim \\mathcal{U}$\nUnlabeled dataset.\n$\\mathcal{D} = \\mathcal{X} \\cup \\mathcal{U}$\nThe entire dataset, including both labeled and unlabeled examples.\n$\\mathbf{x}$\nAny sample which can be either labeled or unlabeled.\n$\\bar{\\mathbf{x}}$\n$\\mathbf{x}$ with augmentation applied.\n$\\mathbf{x}_i$\nThe $i$-th sample.\n$\\mathcal{L}$, $\\mathcal{L}_s$, $\\mathcal{L}_u$\nLoss, supervised loss, and unsupervised loss.\n$\\mu(t)$\nThe unsupervised loss weight, increasing in time.\n$p(y \\vert \\mathbf{x}), p_\\theta(y \\vert \\mathbf{x})$\nThe conditional probability over the label set given the input.\n$f_\\theta(.)$\nThe implemented neural network with weights $\\theta$, the model that we want to train.\n$\\mathbf{z} = f_\\theta(\\mathbf{x})$\nA vector of logits output by $f$.\n$\\hat{y} = \\text{softmax}(\\mathbf{z})$\nThe predicted label distribution.\n$D[.,.]$\nA distance function between two distributions, such as MSE, cross entropy, KL divergence, etc.\n$\\beta$\nEMA weighting hyperparameter for\nteacher\nmodel weights.\n$\\alpha, \\lambda$\nParameters for MixUp, $\\lambda \\sim \\text{Beta}(\\alpha, \\alpha)$.\n$T$\nTemperature for sharpening the predicted distribution.\n$\\tau$\nA confidence threshold for selecting the qualified prediction.\nHypotheses\n#\nSeveral hypotheses have been discussed in literature to support certain design decisions in semi-supervised learning methods.\nH1:\nSmoothness Assumptions\n: If two data samples are close in a high-density region of the feature space, their labels should be the same or very similar.\nH2:\nCluster Assumptions\n: The feature space has both dense regions and sparse regions. Densely grouped data points naturally form a cluster. Samples in the same cluster are expected to have the same label. This is a small extension of H1.\nH3:\nLow-density Separation Assumptions\n: The decision boundary between classes tends to be located in the sparse, low density regions, because otherwise the decision boundary would cut a high-density cluster into two classes, corresponding to two clusters, which invalidates H1 and H2.\nH4:\nManifold Assumptions\n: The high-dimensional data tends to locate on a low-dimensional manifold. Even though real-world data might be observed in very high dimensions (e.g. such as images of real-world objects/scenes), they actually can be captured by a lower dimensional manifold where certain attributes are captured and similar points are grouped closely (e.g. images of real-world objects/scenes are not drawn from a uniform distribution over all pixel combinations). This enables us to learn a more efficient representation for us to discover and measure similarity between unlabeled data points. This is also the foundation for representation learning. [see\na helpful link\n].\nConsistency Regularization\n#\nConsistency Regularization\n, also known as\nConsistency Training\n, assumes that randomness within the neural network (e.g. with Dropout) or data augmentation transformations should not modify model predictions given the same input. Every method in this section has a consistency regularization loss as $\\mathcal{L}_u$.\nThis idea has been adopted in several\nself-supervised\nlearning\nmethods, such as SimCLR, BYOL, SimCSE, etc. Different augmented versions of the same sample should result in the same representation.\nCross-view training\nin language modeling and multi-view learning in self-supervised learning all share the same motivation.\nΠ-model\n#\nOverview of the Π-model. Two versions of the same input with different stochastic augmentation and dropout masks pass through the network and the outputs are expected to be consistent. (Image source:\nLaine & Aila (2017)\n)\nSajjadi et al. (2016)\nproposed an unsupervised learning loss to minimize the difference between two passes through the network with stochastic transformations (e.g. dropout, random max-pooling) for the same data point. The label is not explicitly used, so the loss can be applied to unlabeled dataset.\nLaine & Aila (2017)\nlater coined the name,\nΠ-Model\n, for such a setup.\n$$\n\\mathcal{L}_u^\\Pi = \\sum_{\\mathbf{x} \\in \\mathcal{D}} \\text{MSE}(f_\\theta(\\mathbf{x}), f'_\\theta(\\mathbf{x}))\n$$\nwhere $f’$ is the same neural network with different stochastic augmentation or dropout masks applied. This loss utilizes the entire dataset.\nTemporal ensembling\n#\nOverview of Temporal Ensembling. The per-sample EMA label prediction is the learning target. (Image source:\nLaine & Aila (2017)\n)\nΠ-model requests the network to run two passes per sample, doubling the computation cost. To reduce the cost,\nTemporal Ensembling\n(\nLaine & Aila 2017\n) maintains an exponential moving average (EMA) of the model prediction in time per training sample $\\tilde{\\mathbf{z}}_i$ as the learning target, which is only evaluated and updated once per epoch. Because the ensemble output $\\tilde{\\mathbf{z}}_i$ is initialized to $\\mathbf{0}$, it is normalized by $(1-\\alpha^t)$ to correct this startup bias. Adam optimizer has such\nbias correction\nterms for the same reason.\n$$\n\\tilde{\\mathbf{z}}^{(t)}_i = \\frac{\\alpha \\tilde{\\mathbf{z}}^{(t-1)}_i + (1-\\alpha) \\mathbf{z}_i}{1-\\alpha^t}\n$$\nwhere $\\tilde{\\mathbf{z}}^{(t)}$ is the ensemble prediction at epoch $t$ and $\\mathbf{z}_i$ is the model prediction in the current round. Note that since $\\tilde{\\mathbf{z}}^{(0)} = \\mathbf{0}$, with correction, $\\tilde{\\mathbf{z}}^{(1)}$ is simply equivalent to $\\mathbf{z}_i$ at epoch 1.\nMean teachers\n#\nOverview of the Mean Teacher framework. (Image source:\nTarvaninen & Valpola, 2017\n)\nTemporal Ensembling keeps track of an EMA of label predictions for each training sample as a learning target. However, this label prediction only changes\nevery epoch\n, making the approach clumsy when the training dataset is large.\nMean Teacher\n(\nTarvaninen & Valpola, 2017\n) is proposed to overcome the slowness of target update by tracking the moving average of model weights instead of model outputs. Let’s call the original model with weights $\\theta$ as the\nstudent\nmodel and the model with moving averaged weights $\\theta’$ across consecutive student models as the\nmean teacher\n: $\\theta’ \\gets \\beta \\theta’ + (1-\\beta)\\theta$\nThe consistency regularization loss is the distance between predictions by the student and teacher and the student-teacher gap should be minimized. The mean teacher is expected to provide more accurate predictions than the student. It got confirmed in the empirical experiments, as shown in\nClassification error on SVHN of Mean Teacher and the Π Model. The mean teacher (in orange) has better performance than the student model (in blue). (Image source:\nTarvaninen & Valpola, 2017\n)\nAccording to their ablation studies,\nInput augmentation (e.g. random flips of input images, Gaussian noise) or student model dropout is necessary for good performance. Dropout is not needed on the teacher model.\nThe performance is sensitive to the EMA decay hyperparameter $\\beta$. A good strategy is to use a small $\\beta=0.99$ during the ramp up stage and a larger $\\beta=0.999$ in the later stage when the student model improvement slows down.\nThey found that MSE as the consistency cost function performs better than other cost functions like KL divergence.\nNoisy samples as learning targets\n#\nSeveral recent consistency training methods learn to minimize prediction difference between the original unlabeled sample and its corresponding augmented version. It is quite similar to the Π-model but the consistency regularization loss is\nonly\napplied to the unlabeled data.\nConsistency training with noisy samples.\nAdversarial Training (\nGoodfellow et al. 2014\n) applies adversarial noise onto the input and trains the model to be robust to such adversarial attack. The setup works in supervised learning,\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{adv}(\\mathbf{x}^l, \\theta) &= D[q(y\\mid \\mathbf{x}^l), p_\\theta(y\\mid \\mathbf{x}^l + r_\\text{adv})] \\\\\nr_\\text{adv} &= {\\arg\\max}_{r; \\|r\\| \\leq \\epsilon} D[q(y\\mid \\mathbf{x}^l), p_\\theta(y\\mid \\mathbf{x}^l + r_\\text{adv})] \\\\\nr_\\text{adv} &\\approx \\epsilon \\frac{g}{\\|g\\|_2} \\approx \\epsilon\\text{sign}(g)\\quad\\text{where }g = \\nabla_{r} D[y, p_\\theta(y\\mid \\mathbf{x}^l + r)]\n\\end{aligned}\n$$\nwhere $q(y \\mid \\mathbf{x}^l)$ is the true distribution, approximated by one-hot encoding of the ground truth label, $y$. $p_\\theta(y \\mid \\mathbf{x}^l)$ is the model prediction. $D[.,.]$ is a distance function measuring the divergence between two distributions.\nVirtual Adversarial Training\n(\nVAT\n;\nMiyato et al. 2018\n) extends the idea to work in semi-supervised learning. Because $q(y \\mid \\mathbf{x}^l)$ is unknown, VAT replaces it with the current model prediction for the original input with the current weights $\\hat{\\theta}$. Note that $\\hat{\\theta}$ is a fixed copy of model weights, so there is no gradient update on $\\hat{\\theta}$.\n$$\n\\begin{aligned}\n\\mathcal{L}_u^\\text{VAT}(\\mathbf{x}, \\theta) &= D[p_{\\hat{\\theta}}(y\\mid \\mathbf{x}), p_\\theta(y\\mid \\mathbf{x} + r_\\text{vadv})] \\\\\nr_\\text{vadv} &= {\\arg\\max}_{r; \\|r\\| \\leq \\epsilon} D[p_{\\hat{\\theta}}(y\\mid \\mathbf{x}), p_\\theta(y\\mid \\mathbf{x} + r)]\n\\end{aligned}\n$$\nThe VAT loss applies to both labeled and unlabeled samples. It is a negative smoothness measure of the current model’s prediction manifold at each data point. The optimization of such loss motivates the manifold to be smoother.\nInterpolation Consistency Training\n(\nICT\n;\nVerma et al. 2019\n) enhances the dataset by adding more interpolations of data points and expects the model prediction to be consistent with interpolations of the corresponding labels. MixUp (\nZheng et al. 2018\n) operation mixes two images via a simple weighted sum and combines it with label smoothing. Following the idea of MixUp, ICT expects the prediction model to produce a label on a mixup sample to match the interpolation of predictions of corresponding inputs:\n$$\n\\begin{aligned}\n\\text{mixup}_\\lambda (\\mathbf{x}_i, \\mathbf{x}_j) &= \\lambda \\mathbf{x}_i + (1-\\lambda)\\mathbf{x}_j \\\\\np(\\text{mixup}_\\lambda (y \\mid \\mathbf{x}_i, \\mathbf{x}_j)) &\\approx \\lambda p(y \\mid \\mathbf{x}_i) + (1-\\lambda) p(y \\mid \\mathbf{x}_j)\n\\end{aligned}\n$$\nwhere $\\theta’$ is a moving average of $\\theta$, which is a\nmean teacher\n.\nOverview of Interpolation Consistency Training. MixUp is applied to produce more interpolated samples with interpolated labels as learning targets. (Image source:\nVerma et al. 2019\n)\nBecause the probability of two randomly selected unlabeled samples belonging to different classes is high (e.g. There are 1000 object classes in ImageNet), the interpolation by applying a mixup between two random unlabeled samples is likely to happen around the decision boundary. According to the low-density separation\nassumptions\n, the decision boundary tends to locate in the low density regions.\n$$\n\\mathcal{L}^\\text{ICT}_{u} = \\mathbb{E}_{\\mathbf{u}_i, \\mathbf{u}_j \\sim \\mathcal{U}} \\mathbb{E}_{\\lambda \\sim \\text{Beta}(\\alpha, \\alpha)} D[p_\\theta(y \\mid \\text{mixup}_\\lambda (\\mathbf{u}_i, \\mathbf{u}_j)), \\text{mixup}_\\lambda(p_{\\theta’}(y \\mid \\mathbf{u}_i), p_{\\theta'}(y \\mid \\mathbf{u}_j)]\n$$\nwhere $\\theta’$ is a moving average of $\\theta$.\nSimilar to VAT,\nUnsupervised Data Augmentation\n(\nUDA\n;\nXie et al. 2020\n) learns to predict the same output for an unlabeled example and the augmented one. UDA especially focuses on studying how the\n“quality”\nof noise can impact the semi-supervised learning performance with consistency training. It is crucial to use advanced data augmentation methods for producing meaningful and effective noisy samples. Good data augmentation should produce valid (i.e. does not change the label) and diverse noise, and carry targeted inductive biases.\nFor images, UDA adopts RandAugment (\nCubuk et al. 2019\n) which uniformly samples augmentation operations available in\nPIL\n, no learning or optimization, so it is much cheaper than AutoAugment.\nComparison of various semi-supervised learning methods on CIFAR-10 classification. Fully supervised Wide-ResNet-28-2 and PyramidNet+ShakeDrop have an error rate of **5.4** and **2.7** respectively when trained on 50,000 examples without RandAugment. (Image source:\nXie et al. 2020\n)\nFor language, UDA combines back-translation and TF-IDF based word replacement. Back-translation preserves the high-level meaning but may not retain certain words, while TF-IDF based word replacement drops uninformative words with low TF-IDF scores. In the experiments on language tasks, they found UDA to be complementary to transfer learning and representation learning; For example, BERT fine-tuned (i.e. $\\text{BERT}_\\text{FINETUNE}$ in Fig. 8.) on in-domain unlabeled data can further improve the performance.\nComparison of UDA with different initialization configurations on various text classification tasks. (Image source:\nXie et al. 2020\n)\nWhen calculating $\\mathcal{L}_u$, UDA found two training techniques to help improve the results.\nLow confidence masking\n: Mask out examples with low prediction confidence if lower than a threshold $\\tau$.\nSharpening prediction distribution\n: Use a low temperature $T$ in softmax to sharpen the predicted probability distribution.\nIn-domain data filtration\n: In order to extract more in-domain data from a large out-of-domain dataset, they trained a classifier to predict in-domain labels and then retain samples with high confidence predictions as in-domain candidates.\n$$\n\\begin{aligned}\n&\\mathcal{L}_u^\\text{UDA} = \\mathbb{1}[\\max_{y'} p_{\\hat{\\theta}}(y'\\mid \\mathbf{x}) > \\tau ] \\cdot D[p^\\text{(sharp)}_{\\hat{\\theta}}(y \\mid \\mathbf{x}; T), p_\\theta(y \\mid \\bar{\\mathbf{x}})] \\\\\n&\\text{where } p_{\\hat{\\theta}}^\\text{(sharp)}(y \\mid \\mathbf{x}; T) = \\frac{\\exp(z^{(y)} / T)}{ \\sum_{y'} \\exp(z^{(y')} / T) }\n\\end{aligned}\n$$\nwhere $\\hat{\\theta}$ is a fixed copy of model weights, same as in VAT, so no gradient update, and $\\bar{\\mathbf{x}}$ is the augmented data point. $\\tau$ is the prediction confidence threshold and $T$ is the distribution sharpening temperature.\nPseudo Labeling\n#\nPseudo Labeling\n(\nLee 2013\n) assigns fake labels to unlabeled samples based on the maximum softmax probabilities predicted by the current model and then trains the model on both labeled and unlabeled samples simultaneously in a pure supervised setup.\nWhy could pseudo labels work? Pseudo label is in effect equivalent to\nEntropy Regularization\n(\nGrandvalet & Bengio 2004\n), which minimizes the conditional entropy of class probabilities for unlabeled data to favor low density separation between classes. In other words, the predicted class probabilities is in fact a measure of class overlap, minimizing the entropy is equivalent to reduced class overlap and thus low density separation.\nt-SNE visualization of outputs on MNIST test set by models training (a) without and (b) with pseudo labeling on 60000 unlabeled samples, in addition to 600 labeled data. Pseudo labeling leads to better segregation in the learned embedding space. (Image source:\nLee 2013\n)\nTraining with pseudo labeling naturally comes as an iterative process. We refer to the model that produces pseudo labels as teacher and the model that learns with pseudo labels as student.\nLabel propagation\n#\nLabel Propagation\n(\nIscen et al. 2019\n) is an idea to construct a similarity graph among samples based on feature embedding. Then the pseudo labels are “diffused” from known samples to unlabeled ones where the propagation weights are proportional to pairwise similarity scores in the graph. Conceptually it is similar to a k-NN classifier and both suffer from the problem of not scaling up well with a large dataset.\nIllustration of how Label Propagation works. (Image source:\nIscen et al. 2019\n)\nSelf-Training\n#\nSelf-Training\nis not a new concept (\nScudder 1965\n,\nNigram & Ghani CIKM 2000\n). It is an iterative algorithm, alternating between the following two steps until every unlabeled sample has a label assigned:\nInitially it builds a classifier on labeled data.\nThen it uses this classifier to predict labels for the unlabeled data and converts the most confident ones into labeled samples.\nXie et al. (2020)\napplied self-training in deep learning and achieved great results. On the ImageNet classification task, they first trained an EfficientNet (\nTan & Le 2019\n) model as teacher to generate pseudo labels for 300M unlabeled images and then trained a larger EfficientNet as student to learn with both true labeled and pseudo labeled images. One critical element in their setup is to have\nnoise\nduring student model training but have no noise for the teacher to produce pseudo labels. Thus their method is called\nNoisy Student\n. They applied stochastic depth (\nHuang et al. 2016\n), dropout and RandAugment to noise the student. Noise is important for the student to perform better than the teacher. The added noise has a compound effect to encourage the model’s decision making frontier to be smooth, on both labeled and unlabeled data.\nA few other important technical configs in noisy student self-training are:\nThe student model should be sufficiently large (i.e. larger than the teacher) to fit more data.\nNoisy student should be paired with data balancing, especially important to balance the number of pseudo labeled images in each class.\nSoft pseudo labels work better than hard ones.\nNoisy student also improves adversarial robustness against an FGSM (Fast Gradient Sign Attack = The attack uses the gradient of the loss w.r.t the input data and adjusts the input data to maximize the loss) attack though the model is not optimized for adversarial robustness.\nSentAugment, proposed by\nDu et al. (2020)\n, aims to solve the problem when there is not enough in-domain unlabeled data for self-training in the language domain. It relies on sentence embedding to find unlabeled in-domain samples from a large corpus and uses the retrieved sentences for self-training.\nReducing confirmation bias\n#\nConfirmation bias is a problem with incorrect pseudo labels provided by an imperfect teacher model. Overfitting to wrong labels may not give us a better student model.\nTo reduce confirmation bias,\nArazo et al. (2019)\nproposed two techniques. One is to adopt MixUp with soft labels. Given two samples, $(\\mathbf{x}_i, \\mathbf{x}_j)$ and their corresponding true or pseudo labels $(y_i, y_j)$, the interpolated label equation can be translated to a cross entropy loss with softmax outputs:\n$$\n\\begin{aligned}\n&\\bar{\\mathbf{x}} = \\lambda \\mathbf{x}_i + (1-\\lambda) \\mathbf{x}_j \\\\\n&\\bar{y} = \\lambda y_i + (1-\\lambda) y_j \\Leftrightarrow\n\\mathcal{L} = \\lambda [y_i^\\top \\log f_\\theta(\\bar{\\mathbf{x}})] + (1-\\lambda) [y_j^\\top \\log f_\\theta(\\bar{\\mathbf{x}})]\n\\end{aligned}\n$$\nMixup is insufficient if there are too few labeled samples. They further set a minimum number of labeled samples in every mini batch by oversampling the labeled samples. This works better than upweighting labeled samples, because it leads to more frequent updates rather than few updates of larger magnitude which could be less stable. Like consistency regularization, data augmentation and dropout are also important for pseudo labeling to work well.\nMeta Pseudo Labels\n(\nPham et al. 2021\n) adapts the teacher model constantly with the feedback of how well the student performs on the labeled dataset. The teacher and the student are trained in parallel, where the teacher learns to generate better pseudo labels and the student learns from the pseudo labels.\nLet the teacher and student model weights be $\\theta_T$ and $\\theta_S$, respectively. The student model’s loss on the labeled samples is defined as a function $\\theta^\\text{PL}_S(.)$ of $\\theta_T$ and we would like to minimize this loss by optimizing the teacher model accordingly.\n$$\n\\begin{aligned}\n\\min_{\\theta_T} &\\mathcal{L}_s(\\theta^\\text{PL}_S(\\theta_T)) = \\min_{\\theta_T} \\mathbb{E}_{(\\mathbf{x}^l, y) \\in \\mathcal{X}} \\text{CE}[y, f_{\\theta_S}(\\mathbf{x}^l)] \\\\\n\\text{where } &\\theta^\\text{PL}_S(\\theta_T)\n= \\arg\\min_{\\theta_S} \\mathcal{L}_u (\\theta_T, \\theta_S)\n= \\arg\\min_{\\theta_S} \\mathbb{E}_{\\mathbf{u} \\sim \\mathcal{U}} \\text{CE}[(f_{\\theta_T}(\\mathbf{u}), f_{\\theta_S}(\\mathbf{u}))]\n\\end{aligned}\n$$\nHowever, it is not trivial to optimize the above equation. Borrowing the idea of\nMAML\n, it approximates the multi-step $\\arg\\min_{\\theta_S}$ with the one-step gradient update of $\\theta_S$,\n$$\n\\begin{aligned}\n\\theta^\\text{PL}_S(\\theta_T) &\\approx \\theta_S - \\eta_S \\cdot \\nabla_{\\theta_S} \\mathcal{L}_u(\\theta_T, \\theta_S) \\\\\n\\min_{\\theta_T} \\mathcal{L}_s (\\theta^\\text{PL}_S(\\theta_T)) &\\approx \\min_{\\theta_T} \\mathcal{L}_s \\big( \\theta_S - \\eta_S \\cdot \\nabla_{\\theta_S} \\mathcal{L}_u(\\theta_T, \\theta_S) \\big)\n\\end{aligned}\n$$\nWith soft pseudo labels, the above objective is differentiable. But if using hard pseudo labels, it is not differentiable and thus we need to use RL, e.g. REINFORCE.\nThe optimization procedure is alternative between training two models:\nStudent model update\n: Given a batch of unlabeled samples $\\{ \\mathbf{u} \\}$, we generate pseudo labels by $f_{\\theta_T}(\\mathbf{u})$ and optimize $\\theta_S$ with one step SGD: $\\theta’_S = \\color{green}{\\theta_S - \\eta_S \\cdot \\nabla_{\\theta_S} \\mathcal{L}_u(\\theta_T, \\theta_S)}$.\nTeacher model update\n: Given a batch of labeled samples $\\{(\\mathbf{x}^l, y)\\}$, we reuse the student’s update to optimize $\\theta_T$: $\\theta’_T = \\theta_T - \\eta_T \\cdot \\nabla_{\\theta_T} \\mathcal{L}_s ( \\color{green}{\\theta_S - \\eta_S \\cdot \\nabla_{\\theta_S} \\mathcal{L}_u(\\theta_T, \\theta_S)} )$. In addition, the UDA objective is applied to the teacher model to incorporate consistency regularization.\nComparison of Meta Pseudo Labels with other semi- or self-supervised learning methods on image classification tasks. (Image source:\nPham et al. 2021\n)\nPseudo Labeling with Consistency Regularization\n#\nIt is possible to combine the above two approaches together, running semi-supervised learning with both pseudo labeling and consistency training.\nMixMatch\n#\nMixMatch\n(\nBerthelot et al. 2019\n), as a holistic approach to semi-supervised learning, utilizes unlabeled data by merging the following techniques:\nConsistency regularization\n: Encourage the model to output the same predictions on perturbed unlabeled samples.\nEntropy minimization\n: Encourage the model to output confident predictions on unlabeled data.\nMixUp\naugmentation: Encourage the model to have linear behaviour between samples.\nGiven a batch of labeled data $\\mathcal{X}$ and unlabeled data $\\mathcal{U}$, we create augmented versions of them via $\\text{MixMatch}(.)$, $\\bar{\\mathcal{X}}$ and $\\bar{\\mathcal{U}}$, containing augmented samples and guessed labels for unlabeled examples.\n$$\n\\begin{aligned}\n\\bar{\\mathcal{X}}, \\bar{\\mathcal{U}} &= \\text{MixMatch}(\\mathcal{X}, \\mathcal{U}, T, K, \\alpha) \\\\\n\\mathcal{L}^\\text{MM}_s &= \\frac{1}{\\vert \\bar{\\mathcal{X}} \\vert} \\sum_{(\\bar{\\mathbf{x}}^l, y)\\in \\bar{\\mathcal{X}}} D[y, p_\\theta(y \\mid \\bar{\\mathbf{x}}^l)] \\\\\n\\mathcal{L}^\\text{MM}_u &= \\frac{1}{L\\vert \\bar{\\mathcal{U}} \\vert} \\sum_{(\\bar{\\mathbf{u}}, \\hat{y})\\in \\bar{\\mathcal{U}}} \\| \\hat{y} - p_\\theta(y \\mid \\bar{\\mathbf{u}}) \\|^2_2 \\\\\n\\end{aligned}\n$$\nwhere $T$ is the sharpening temperature to reduce the guessed label overlap; $K$ is the number of augmentations generated per unlabeled example; $\\alpha$ is the parameter in MixUp.\nFor each $\\mathbf{u}$, MixMatch generates $K$ augmentations, $\\bar{\\mathbf{u}}^{(k)} = \\text{Augment}(\\mathbf{u})$ for $k=1, \\dots, K$ and the pseudo label is guessed based on the average: $\\hat{y} = \\frac{1}{K} \\sum_{k=1}^K p_\\theta(y \\mid \\bar{\\mathbf{u}}^{(k)})$.\nThe process of \"label guessing\" in MixMatch: averaging $K$ augmentations, correcting the predicted marginal distribution and finally sharpening the distribution. (Image source:\nBerthelot et al. 2019\n)\nAccording to their ablation studies, it is critical to have MixUp especially on the unlabeled data. Removing temperature sharpening on the pseudo label distribution hurts the performance quite a lot. Average over multiple augmentations for label guessing is also necessary.\nReMixMatch\n(\nBerthelot et al. 2020\n) improves MixMatch by introducing two new mechanisms:\nIllustration of two improvements introduced in ReMixMatch over MixMatch. (Image source:\nBerthelot et al. 2020\n)\nDistribution alignment.\nIt encourages the marginal distribution $p(y)$ to be close to the marginal distribution of the ground truth labels. Let $p(y)$ be the class distribution in the true labels and $\\tilde{p}(\\hat{y})$ be a running average of the predicted class distribution among the unlabeled data. The model prediction on an unlabeled sample $p_\\theta(y \\vert \\mathbf{u})$ is normalized to be $\\text{Normalize}\\big( \\frac{p_\\theta(y \\vert \\mathbf{u}) p(y)}{\\tilde{p}(\\hat{y})} \\big)$ to match the true marginal distribution.\nNote that entropy minimization is not a useful objective if the marginal distribution is not uniform.\nI do feel the assumption that the class distributions on the labeled and unlabeled data should match is too strong and not necessarily to be true in the real-world setting.\nAugmentation anchoring\n. Given an unlabeled sample, it first generates an “anchor” version with weak augmentation and then averages $K$ strongly augmented versions using CTAugment (Control Theory Augment). CTAugment only samples augmentations that keep the model predictions within the network tolerance.\nThe ReMixMatch loss is a combination of several terms,\na supervised loss with data augmentation and MixUp applied;\nan unsupervised loss with data augmentation and MixUp applied, using pseudo labels as targets;\na CE loss on a single heavily-augmented unlabeled image without MixUp;\na\nrotation\nloss as in self-supervised learning.\nDivideMix\n#\nDivideMix\n(\nJunnan Li et al. 2020\n) combines semi-supervised learning with Learning with noisy labels (LNL). It models the per-sample loss distribution via a\nGMM\nto dynamically divide the training data into a labeled set with clean examples and an unlabeled set with noisy ones. Following the idea in\nArazo et al. 2019\n, they fit a two-component GMM on the per-sample cross entropy loss $\\ell_i = y_i^\\top \\log f_\\theta(\\mathbf{x}_i)$. Clean samples are expected to get lower loss faster than noisy samples. The component with smaller mean is the cluster corresponding to clean labels and let’s denote it as $c$. If the GMM posterior probability $w_i = p_\\text{GMM}(c \\mid \\ell_i)$ (i.e. the probability of the sampling belonging to the clean sample set) is larger than the threshold $\\tau$, this sample is considered as a clean sample and otherwise a noisy one.\nThe data clustering step is named\nco-divide\n. To avoid confirmation bias, DivideMix simultaneously trains two diverged networks where each network uses the dataset division from the other network; e.g. thinking about how Double Q Learning works.\nDivideMix trains two networks independently to reduce confirmation bias. They run co-divide, co-refinement, and co-guessing together. (Image source:\nJunnan Li et al. 2020\n)\nCompared to MixMatch, DivideMix has an additional\nco-divide\nstage for handling noisy samples, as well as the following improvements during training:\nLabel co-refinement\n: It linearly combines the ground-truth label $y_i$ with the network’s prediction $\\hat{y}_i$, which is averaged across multiple augmentations of $\\mathbf{x}_i$, guided by the clean set probability $w_i$ produced by the other network.\nLabel co-guessing\n: It averages the predictions from two models for unlabelled data samples.\nThe algorithm of DivideMix. (Image source:\nJunnan Li et al. 2020\n)\nFixMatch\n#\nFixMatch\n(\nSohn et al. 2020\n) generates pseudo labels on unlabeled samples with weak augmentation and only keeps predictions with high confidence. Here both weak augmentation and high confidence filtering help produce high-quality trustworthy pseudo label targets. Then FixMatch learns to predict these pseudo labels given a heavily-augmented sample.\nIllustration of how FixMatch works. (Image source:\nSohn et al. 2020\n)\n$$\n\\begin{aligned}\n\\mathcal{L}_s &= \\frac{1}{B} \\sum^B_{b=1} \\text{CE}[y_b, p_\\theta(y \\mid \\mathcal{A}_\\text{weak}(\\mathbf{x}_b))] \\\\\n\\mathcal{L}_u &= \\frac{1}{\\mu B} \\sum_{b=1}^{\\mu B} \\mathbb{1}[\\max(\\hat{y}_b) \\geq \\tau]\\;\\text{CE}(\\hat{y}_b, p_\\theta(y \\mid \\mathcal{A}_\\text{strong}(\\mathbf{u}_b)))\n\\end{aligned}\n$$\nwhere $\\hat{y}_b$ is the pseudo label for an unlabeled example; $\\mu$ is a hyperparameter that determines the relative sizes of $\\mathcal{X}$ and $\\mathcal{U}$.\nWeak augmentation $\\mathcal{A}_\\text{weak}(.)$: A standard flip-and-shift augmentation\nStrong augmentation $\\mathcal{A}_\\text{strong}(.)$ : AutoAugment, Cutout, RandAugment, CTAugment\nPerformance of FixMatch and several other semi-supervised learning methods on image classification tasks. (Image source:\nSohn et al. 2020\n)\nAccording to the ablation studies of FixMatch,\nSharpening the predicted distribution with a temperature parameter $T$ does not have a significant impact when the threshold $\\tau$ is used.\nCutout and CTAugment as part of strong augmentations are necessary for good performance.\nWhen the weak augmentation for label guessing is replaced with strong augmentation, the model diverges early in training. If discarding weak augmentation completely, the model overfit the guessed labels.\nUsing weak instead of strong augmentation for pseudo label prediction leads to unstable performance. Strong data augmentation is critical.\nCombined with Powerful Pre-Training\n#\nIt is a common paradigm, especially in language tasks, to first pre-train a task-agnostic model on a large unsupervised data corpus via self-supervised learning and then fine-tune it on the downstream task with a small labeled dataset. Research has shown that we can obtain extra gain if combining semi-supervised learning with pretraining.\nZoph et al. (2020)\nstudied to what degree\nself-training\ncan work better than pre-training. Their experiment setup was to use ImageNet for pre-training or self-training to improve COCO. Note that when using ImageNet for self-training, it discards labels and only uses ImageNet samples as unlabeled data points.\nHe et al. (2018)\nhas demonstrated that ImageNet classification pre-training does not work well if the downstream task is very different, such as object detection.\nThe effect of (a) data augment (from weak to strong) and (b) the labeled dataset size on the object detection performance. In the legend: `Rand Init` refers to a model initialized w/ random weights; `ImageNet` is initialized with a pre-trained checkpoint at 84.5% top-1 ImageNet accuracy; `ImageNet++` is initialized with a checkpoint with a higher accuracy 86.9%. (Image source:\nZoph et al. 2020\n)\nTheir experiments demonstrated a series of interesting findings:\nThe effectiveness of pre-training diminishes with more labeled samples available for the downstream task. Pre-training is helpful in the low-data regimes (20%) but neutral or harmful in the high-data regime.\nSelf-training helps in high data/strong augmentation regimes, even when pre-training hurts.\nSelf-training can bring in additive improvement on top of pre-training, even using the same data source.\nSelf-supervised pre-training (e.g. via SimCLR) hurts the performance in a high data regime, similar to how supervised pre-training does.\nJoint-training supervised and self-supervised objectives help resolve the mismatch between the pre-training and downstream tasks. Pre-training, joint-training and self-training are all additive.\nNoisy labels or un-targeted labeling (i.e. pre-training labels are not aligned with downstream task labels) is worse than targeted pseudo labeling.\nSelf-training is computationally more expensive than fine-tuning on a pre-trained model.\nChen et al. (2020)\nproposed a three-step procedure to merge the benefits of self-supervised pretraining, supervised fine-tuning and self-training together:\nUnsupervised or self-supervised pretrain a big model.\nSupervised fine-tune it on a few labeled examples. It is important to use a big (deep and wide) neural network.\nBigger models yield better performance with fewer labeled samples.\nDistillation with unlabeled examples by adopting pseudo labels in self-training.\nIt is possible to distill the knowledge from a large model into a small one because the task-specific use does not require extra capacity of the learned representation.\nThe distillation loss is formatted as the following, where the teacher network is fixed with weights $\\hat{\\theta}_T$.\n$$\n \\mathcal{L}_\\text{distill} = - (1-\\alpha) \\underbrace{\\sum_{(\\mathbf{x}^l_i, y_i) \\in \\mathcal{X}} \\big[ \\log p_{\\theta_S}(y_i \\mid \\mathbf{x}^l_i) \\big]}_\\text{Supervised loss} - \\alpha \\underbrace{\\sum_{\\mathbf{u}_i \\in \\mathcal{U}} \\Big[ \\sum_{i=1}^L p_{\\hat{\\theta}_T}(y^{(i)} \\mid \\mathbf{u}_i; T) \\log p_{\\theta_S}(y^{(i)} \\mid \\mathbf{u}_i; T) \\Big]}_\\text{Distillation loss using unlabeled data}\n $$\nA semi-supervised learning framework leverages unlabeled data corpus by (Left) task-agnostic unsupervised pretraining and (Right) task-specific self-training and distillation. (Image source:\nChen et al. 2020\n)\nThey experimented on the ImageNet classification task. The self-supervised pre-training uses SimCLRv2, a directly improved version of\nSimCLR\n. Observations in their empirical studies confirmed several learnings, aligned with\nZoph et al. 2020\n:\nBigger models are more label-efficient;\nBigger/deeper project heads in SimCLR improve representation learning;\nDistillation using unlabeled data improves semi-supervised learning.\nComparison of performance by SimCLRv2 + semi-supervised distillation on ImageNet classification. (Image source:\nChen et al. 2020\n)\n💡 Quick summary of common themes among recent semi-supervised learning methods, many aiming to reduce confirmation bias:\nApply valid and diverse noise to samples by advanced data augmentation methods.\nWhen dealing with images, MixUp is an effective augmentation. Mixup could work on language too, resulting in a small incremental improvement (\nGuo et al. 2019\n).\nSet a threshold and discard pseudo labels with low confidence.\nSet a minimum number of labeled samples per mini-batch.\nSharpen the pseudo label distribution to reduce the class overlap.\nCitation\n#\nCited as:\nWeng, Lilian. (Dec 2021). Learning with not enough data part 1: semi-supervised learning. Lil’Log. https://lilianweng.github.io/posts/2021-12-05-semi-supervised/.\nOr\n@article{weng2021semi,\n title = \"Learning with not Enough Data Part 1: Semi-Supervised Learning\",\n author = \"Weng, Lilian\",\n journal = \"lilianweng.github.io\",\n year = \"2021\",\n month = \"Dec\",\n url = \"https://lilianweng.github.io/posts/2021-12-05-semi-supervised/\"\n}\nReferences\n#\n[1] Ouali, Hudelot & Tami.\n“An Overview of Deep Semi-Supervised Learning”\narXiv preprint arXiv:2006.05278 (2020).\n[2] Sajjadi, Javanmardi & Tasdizen\n“Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning.”\narXiv preprint arXiv:1606.04586 (2016).\n[3] Pham et al.\n“Meta Pseudo Labels.”\nCVPR 2021.\n[4] Laine & Aila.\n“Temporal Ensembling for Semi-Supervised Learning”\nICLR 2017.\n[5] Tarvaninen & Valpola.\n“Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.”\nNeuriPS 2017\n[6] Xie et al.\n“Unsupervised Data Augmentation for Consistency Training.”\nNeuriPS 2020.\n[7] Miyato et al.\n“Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning.”\nIEEE transactions on pattern analysis and machine intelligence 41.8 (2018).\n[8] Verma et al.\n“Interpolation consistency training for semi-supervised learning.”\nIJCAI 2019\n[9] Lee.\n“Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.”\nICML 2013 Workshop: Challenges in Representation Learning.\n[10] Iscen et al.\n“Label propagation for deep semi-supervised learning.”\nCVPR 2019.\n[11] Xie et al.\n“Self-training with Noisy Student improves ImageNet classification”\nCVPR 2020.\n[12] Jingfei Du et al.\n“Self-training Improves Pre-training for Natural Language Understanding.”\n2020\n[13] Iscen et al.\n“Label propagation for deep semi-supervised learning.”\nCVPR 2019\n[14] Arazo et al.\n“Pseudo-labeling and confirmation bias in deep semi-supervised learning.”\nIJCNN 2020.\n[15] Berthelot et al.\n“MixMatch: A holistic approach to semi-supervised learning.”\nNeuriPS 2019\n[16] Berthelot et al.\n“ReMixMatch: Semi-supervised learning with distribution alignment and augmentation anchoring.”\nICLR 2020\n[17] Sohn et al.\n“FixMatch: Simplifying semi-supervised learning with consistency and confidence.”\nCVPR 2020\n[18] Junnan Li et al.\n“DivideMix: Learning with Noisy Labels as Semi-supervised Learning.”\n2020 [\ncode\n]\n[19] Zoph et al.\n“Rethinking pre-training and self-training.”\n2020.\n[20] Chen et al.\n“Big Self-Supervised Models are Strong Semi-Supervised Learners”\n2020", "metadata": { "title": "Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Log", "description": "When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.", "keywords": "data, unsupervised-learning", "author": "Lilian Weng", "og:title": "Learning with not Enough Data Part 1: Semi-Supervised Learning", "og:description": "When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.", "og:type": "article", "og:url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/", "twitter:card": "summary", "twitter:title": "Learning with not Enough Data Part 1: Semi-Supervised Learning", "twitter:description": "When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.", "article:section": "posts", "article:published_time": "2021-12-05T00:00:00+00:00", "article:modified_time": "2021-12-05T00:00:00+00:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/76583c4985bd/code/body-000.txt", "chars": 283, "language": "text" } ], "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/PI-model.png", "bytes": 77905, "reused": false, "local_path": "attachments/76583c4985bd/images/body-22f8744911da963ff507.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/PI-model.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/temperal-ensembling.png", "bytes": 83063, "reused": false, "local_path": "attachments/76583c4985bd/images/body-2431b04e51e95d44cbfc.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/temperal-ensembling.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/mean-teacher.png", "bytes": 147122, "reused": false, "local_path": "attachments/76583c4985bd/images/body-3704ca968b8b036dd979.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/mean-teacher.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/mean-teacher-results.png", "bytes": 449224, "reused": false, "local_path": "attachments/76583c4985bd/images/body-264593afb504dba40862.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/mean-teacher-results.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/consistency-training-with-noisy-samples.png", "bytes": 270328, "reused": false, "local_path": "attachments/76583c4985bd/images/body-fe1f57c36a0c6b35ff30.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/consistency-training-with-noisy-samples.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/ICT.png", "bytes": 299310, "reused": false, "local_path": "attachments/76583c4985bd/images/body-686784e7314725fbdcc3.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/ICT.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/UDA-image-results.png", "bytes": 375554, "reused": false, "local_path": "attachments/76583c4985bd/images/body-abc3c1e68e088b944f70.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/UDA-image-results.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/UDA-language-results.png", "bytes": 278212, "reused": false, "local_path": "attachments/76583c4985bd/images/body-20f5ad7f538bcfcaa51f.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/UDA-language-results.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/pseudo-label-segregation.png", "bytes": 3492681, "reused": false, "local_path": "attachments/76583c4985bd/images/body-09be84ee222f7edd8341.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/pseudo-label-segregation.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/label-propagation.png", "bytes": 340862, "reused": false, "local_path": "attachments/76583c4985bd/images/body-41918aaaac4a38312f21.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/label-propagation.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/MPL-results.png", "bytes": 373018, "reused": false, "local_path": "attachments/76583c4985bd/images/body-151ce877219f628d821e.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/MPL-results.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/MixMatch.png", "bytes": 130110, "reused": false, "local_path": "attachments/76583c4985bd/images/body-52dcd8696c072fec5989.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/MixMatch.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/ReMixMatch.png", "bytes": 519536, "reused": false, "local_path": "attachments/76583c4985bd/images/body-60ae1da08ae38b4e5477.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/ReMixMatch.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/DivideMix.png", "bytes": 124124, "reused": false, "local_path": "attachments/76583c4985bd/images/body-1f3e79d5973995850e17.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/DivideMix.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/DivideMix-algo.png", "bytes": 548196, "reused": false, "local_path": "attachments/76583c4985bd/images/body-b8bdf2d20d1fcc25c810.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/DivideMix-algo.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/FixMatch.png", "bytes": 218342, "reused": false, "local_path": "attachments/76583c4985bd/images/body-015dfefe2091cea2d5d5.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/FixMatch.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/FixMatch-results.png", "bytes": 306334, "reused": false, "local_path": "attachments/76583c4985bd/images/body-de572b70c4267a43c66a.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/FixMatch-results.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/self-training-pre-training.png", "bytes": 470728, "reused": false, "local_path": "attachments/76583c4985bd/images/body-cf42d45d1104e492f130.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/self-training-pre-training.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/big-self-supervised-model.png", "bytes": 176722, "reused": false, "local_path": "attachments/76583c4985bd/images/body-452ce3c3d7e9a8712c06.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/big-self-supervised-model.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/big-self-supervised-model-results.png", "bytes": 411449, "reused": false, "local_path": "attachments/76583c4985bd/images/body-9d2b0d8db4abe7b366db.png", "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/big-self-supervised-model-results.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://lilianweng.github.io/posts/2019-01-31-lm/", "label": "pre-training LMs", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2019-11-10-self-supervised/", "label": "self-supervised learning", "type": "a" }, { "href": "https://arxiv.org/abs/2006.05278", "label": "this survey", "type": "a" }, { "href": "https://stats.stackexchange.com/questions/66939/what-is-the-manifold-assumption-in-semi-supervised-learning", "label": "a helpful link", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2021-05-31-contrastive/", "label": "learning", "type": "a" }, { "href": "https://arxiv.org/abs/1610.02242", "label": "Laine & Aila (2017)", "type": "a" }, { "href": "https://arxiv.org/abs/1606.04586", "label": "Sajjadi et al. (2016)", "type": "a" }, { "href": "https://stats.stackexchange.com/questions/232741/why-is-it-important-to-include-a-bias-correction-term-for-the-adam-optimizer-for", "label": "bias correction", "type": "a" }, { "href": "https://arxiv.org/abs/1703.01780", "label": "Tarvaninen & Valpola, 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1412.6572", "label": "Goodfellow et al. 2014", "type": "a" }, { "href": "https://arxiv.org/abs/1704.03976", "label": "Miyato et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1903.03825", "label": "Verma et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1710.09412", "label": "Zheng et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1904.12848", "label": "Xie et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1909.13719", "label": "Cubuk et al. 2019", "type": "a" }, { "href": "https://pillow.readthedocs.io/en/stable/", "label": "PIL", "type": "a" }, { "href": "http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.664.3543&rep=rep1&type=pdf", "label": "Lee 2013", "type": "a" }, { "href": "https://papers.nips.cc/paper/2004/hash/96f2b50b5d3613adf9c27049b2a888c7-Abstract.html", "label": "Grandvalet & Bengio 2004", "type": "a" }, { "href": "https://arxiv.org/abs/1904.04717", "label": "Iscen et al. 2019", "type": "a" }, { "href": "https://ieeexplore.ieee.org/document/1053799", "label": "Scudder 1965", "type": "a" }, { "href": "http://www.kamalnigam.com/papers/cotrain-CIKM00.pdf", "label": "Nigram & Ghani CIKM 2000", "type": "a" }, { "href": "https://arxiv.org/abs/1911.04252", "label": "Xie et al. (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/1905.11946", "label": "Tan & Le 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1603.09382", "label": "Huang et al. 2016", "type": "a" }, { "href": "https://arxiv.org/abs/2010.02194", "label": "Du et al. (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/1908.02983", "label": "Arazo et al. (2019)", "type": "a" }, { "href": "https://arxiv.org/abs/2003.10580", "label": "Pham et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/1703.03400", "label": "MAML", "type": "a" }, { "href": "https://arxiv.org/abs/1905.02249", "label": "Berthelot et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1911.09785", "label": "Berthelot et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2002.07394", "label": "Junnan Li et al. 2020", "type": "a" }, { "href": "https://scikit-learn.org/stable/modules/mixture.html", "label": "GMM", "type": "a" }, { "href": "https://arxiv.org/abs/1904.11238", "label": "Arazo et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/2001.07685", "label": "Sohn et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2006.06882", "label": "Zoph et al. (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/1811.08883", "label": "He et al. (2018)", "type": "a" }, { "href": "https://arxiv.org/abs/2006.10029", "label": "Chen et al. (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/1905.08941", "label": "Guo et al. 2019", "type": "a" }, { "href": "https://github.com/LiJunnan1992/DivideMix", "label": "code", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:51:21Z", "source_url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/2006.05278", "bytes": 3172260, "reused": false, "local_path": "papers/repair/85bc9fdf7b6d2f5c5ff6.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1610.02242", "bytes": 322144, "reused": false, "local_path": "papers/repair/1366267456ae3965f8c9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1606.04586", "bytes": 282460, "reused": false, "local_path": "papers/repair/dfcdfea8469d7c081d06.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1703.01780", "bytes": 863822, "reused": false, "local_path": "papers/repair/f5e8782daef1aa339f71.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1412.6572", "bytes": 1036360, "reused": false, "local_path": "papers/repair/1b6f12d7db56a03ce596.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1704.03976", "bytes": 8496008, "reused": false, "local_path": "papers/repair/7c610a3bc37a3c724c6c.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1903.03825", "bytes": 2774709, "reused": false, "local_path": "papers/repair/c0ed786bf3a3169d24a7.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1710.09412", "bytes": 740562, "reused": false, "local_path": "papers/repair/7af2d52fbf38a69ec251.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1904.12848", "bytes": 1625724, "reused": false, "local_path": "papers/repair/17af125202b45606d9ad.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.13719", "bytes": 1263406, "reused": false, "local_path": "papers/repair/c267ff367c4770ddb6c0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1904.04717", "bytes": 626230, "reused": false, "local_path": "papers/repair/ae6e8a0596cf6995095d.pdf" }, { "status": "ok", "url": "http://www.kamalnigam.com/papers/cotrain-CIKM00.pdf", "bytes": 230483, "reused": false, "local_path": "papers/repair/aa557a37f131c5edaca1.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1911.04252", "bytes": 2801395, "reused": false, "local_path": "papers/repair/2d5071bca517ed21302d.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.11946", "bytes": 933600, "reused": false, "local_path": "papers/repair/b9c9141c18b68865cdc7.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1603.09382", "bytes": 989025, "reused": false, "local_path": "papers/repair/531ae1708a14ac8b81dc.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2010.02194", "bytes": 2525939, "reused": false, "local_path": "papers/repair/46cf1aae6a3b935b22db.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1908.02983", "bytes": 904317, "reused": false, "local_path": "papers/repair/4e39b8b3af735acbdef2.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2003.10580", "bytes": 1245388, "reused": false, "local_path": "papers/repair/19c860b2949c2d91d5ec.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1703.03400", "bytes": 4064212, "reused": false, "local_path": "papers/repair/745c6af0eff74ad13278.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.02249", "bytes": 520905, "reused": false, "local_path": "papers/repair/c84670ad345d150685e0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1911.09785", "bytes": 420212, "reused": false, "local_path": "papers/repair/e1bda50022ee2a740e73.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.07394", "bytes": 771769, "reused": false, "local_path": "papers/repair/a688eaa44d9b3d4d3846.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1904.11238", "bytes": 3862031, "reused": false, "local_path": "papers/repair/4f8506d9f9d2559e3846.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2001.07685", "bytes": 770166, "reused": false, "local_path": "papers/repair/f068f1c17b46bc3eed3a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2006.06882", "bytes": 1655938, "reused": false, "local_path": "papers/repair/d7e01a0318a00ae28138.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1811.08883", "bytes": 788571, "reused": false, "local_path": "papers/repair/db0e3c5521e5a1800c77.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2006.10029", "bytes": 823706, "reused": false, "local_path": "papers/repair/558bd6a836d462b86d7a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.08941", "bytes": 345640, "reused": true, "local_path": "papers/repair/646a76cd251b8253359a.pdf" } ], "actual_video_files": [], "body_chars": 39098, "expected_body_images": 20, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/76583c4985bd/pages/source.html", "local_html": "attachments/76583c4985bd/pages/article.html", "body_base_path": "attachments/76583c4985bd/pages/", "body_markdown_path": "attachments/76583c4985bd/pages/article.md" } }, { "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/", "status": "ok", "article_id": "028a8389cdec", "local_path": "series-follow/1602/repair/d2f5acee51aa81de8884.html", "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-workflow.png", "bytes": 251848, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/d81486b940074ca7bee2.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/DBAL-exp.png", "bytes": 255226, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/a6d3abf0912005132083.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-loss-prediction.png", "bytes": 208573, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/533dde5b3386ec869780.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-loss-prediction-exp.png", "bytes": 582217, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/a27ab134b2539e4fa927.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/VAAL.png", "bytes": 545888, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/f2a382c35839b44510af.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/VAAL-exp.png", "bytes": 705319, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/5b0a42592fcac6b2269f.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/MAL.png", "bytes": 412179, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/f70d56c397cd0359db6b.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/MAL-exp.png", "bytes": 264815, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/e108c7cf704e84720825.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/core-sets-exp.png", "bytes": 1047500, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/bd8c66b6e5605f2b18ac.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/BADGE-algo.png", "bytes": 246545, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/57159f6aa6eb14ac2a40.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/label-dispersion-vs-uncertainty.png", "bytes": 217492, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/cad8d44f6e9a7e9a6aef.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/CEAL.png", "bytes": 896102, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/5b32e280369a4496da10.png" } ], "article": { "source_file": "0930", "source_platform": "个人博客", "author": "Lilian Weng / Lil'Log", "title_listed": "Learning with not Enough Data Part 2: Active Learning", "keywords": "主动学习,数据标注,采样", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/", "publish_date": "2022-02", "corpus_stage": "数据处理", "notes": "主动学习系列综述之二", "id": "028a8389cdec", "collected_at": "2026-10-02T14:13:03", "title": "Learning with not Enough Data Part 2: Active Learning | Lil'Log", "body_markdown": "This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.\nNotations\n#\nSymbol\nMeaning\n$K$\nNumber of unique class labels.\n$(\\mathbf{x}^l, y) \\sim \\mathcal{X}, y \\in \\{0, 1\\}^K$\nLabeled dataset. $y$ is a one-hot representation of the true label.\n$\\mathbf{u} \\sim \\mathcal{U}$\nUnlabeled dataset.\n$\\mathcal{D} = \\mathcal{X} \\cup \\mathcal{U}$\nThe entire dataset, including both labeled and unlabeled examples.\n$\\mathbf{x}$\nAny sample which can be either labeled or unlabeled.\n$\\mathbf{x}_i$\nThe $i$-th sample.\n$U(\\mathbf{x})$\nScoring function for active learning selection.\n$P_\\theta(y \\vert \\mathbf{x})$\nA softmax classifier parameterized by $\\theta$.\n$\\hat{y} = \\arg\\max_{y \\in \\mathcal{Y}} P_\\theta(y \\vert \\mathbf{x})$\nThe most confident prediction by the classifier.\n$B$\nLabeling budget (the maximum number of samples to label).\n$b$\nBatch size.\nWhat is Active Learning?\n#\nGiven an unlabeled dataset $\\mathcal{U}$ and a fixed amount of labeling cost $B$, active learning aims to select a subset of $B$ examples from $\\mathcal{U}$ to be labeled such that they can result in maximized improvement in model performance. This is an effective way of learning especially when data labeling is difficult and costly, e.g. medical images. This classical\nsurvey paper\nin 2010 lists many key concepts. While some conventional approaches may not apply to deep learning, discussion in this post mainly focuses on deep neural models and training in batch mode.\nIllustration of a cyclic workflow of active learning, producing better models more efficiently by smartly choosing which samples to label.\nTo simplify the discussion, we assume that the task is a $K$-class classification problem in all the following sections. The model with parameters $\\theta$ outputs a probability distribution over the label candidates, which may or may not be calibrated, $P_\\theta(y \\vert \\mathbf{x})$ and the most likely prediction is $\\hat{y} = \\arg\\max_{y \\in \\mathcal{Y}} P_\\theta(y \\vert \\mathbf{x})$.\nAcquisition Function\n#\nThe process of identifying the most valuable examples to label next is referred to as “sampling strategy” or “query strategy”. The scoring function in the sampling process is named “acquisition function”, denoted as $U(\\mathbf{x})$. Data points with higher scores are expected to produce higher value for model training if they get labeled.\nHere is a list of basic sampling strategies.\nUncertainty Sampling\n#\nUncertainty sampling\nselects examples for which the model produces most uncertain predictions. Given a single model, uncertainty can be estimated by the predicted probabilities, although one common complaint is that deep learning model predictions are often not calibrated and not correlated with true uncertainty well. In fact, deep learning models are often overconfident.\nLeast confident score\n, also known as\nvariation ratio\n: $U(\\mathbf{x}) = 1 - P_\\theta(\\hat{y} \\vert \\mathbf{x})$.\nMargin score\n: $U(\\mathbf{x}) = P_\\theta(\\hat{y}_1 \\vert \\mathbf{x}) - P_\\theta(\\hat{y}_2 \\vert \\mathbf{x})$, where $\\hat{y}_1$ and $\\hat{y}_2$ are the most likely and the second likely predicted labels.\nEntropy\n: $U(\\mathbf{x}) = \\mathcal{H}(P_\\theta(y \\vert \\mathbf{x})) = - \\sum_{y \\in \\mathcal{Y}} P_\\theta(y \\vert \\mathbf{x}) \\log P_\\theta(y \\vert \\mathbf{x})$.\nAnother way to quantify uncertainty is to rely on a committee of expert models, known as Query-By-Committee (QBC). QBC measures uncertainty based on a pool of opinions and thus it is critical to keep a level of disagreement among committee members. Given $C$ models in the committee pool, each parameterized by $\\theta_1, \\dots, \\theta_C$.\nVoter entropy\n: $U(\\mathbf{x}) = \\mathcal{H}(\\frac{V(y)}{C})$, where $V(y)$ counts the number of votes from the committee on the label $y$.\nConsensus entropy\n: $U(\\mathbf{x}) = \\mathcal{H}(P_\\mathcal{C})$, where $P_\\mathcal{C}$ is the prediction averaging across the committee.\nKL divergence\n: $U(\\mathbf{x}) = \\frac{1}{C} \\sum_{c=1}^C D_\\text{KL} (P_{\\theta_c} | P_\\mathcal{C})$\nDiversity Sampling\n#\nDiversity sampling\nintend to find a collection of samples that can well represent the entire data distribution. Diversity is important because the model is expected to work well on any data in the wild, just not on a narrow subset. Selected samples should be representative of the underlying distribution. Common approaches often rely on quantifying the similarity between samples.\nExpected Model Change\n#\nExpected model change\nrefers to the impact that a sample brings onto the model training. The impact can be the influence on the model weights or the improvement over the training loss. A\nlater section\nreviews several works on how to measure model impact triggered by selected data samples.\nHybrid Strategy\n#\nMany methods above are not mutually exclusive. A\nhybrid\nsampling strategy values different attributes of data points, combining different sampling preferences into one. Often we want to select\nuncertain but also highly representative\nsamples.\nDeep Acquisition Function\n#\nMeasuring Uncertainty\n#\nThe model uncertainty is commonly categorized into two buckets (\nDer Kiureghian & Ditlevsen 2009\n,\nKendall & Gal 2017\n):\nAleatoric uncertainty\nis introduced by noise in the data (e.g. sensor data, noise in the measurement process) and it can be input-dependent or input-independent. It is generally considered as irreducible since there is missing information about the ground truth.\nEpistemic uncertainty\nrefers to the uncertainty within the model parameters and therefore we do not know whether the model can best explain the data. This type of uncertainty is theoretically reducible given more data\nEnsemble and Approximated Ensemble\n#\nThere is a long tradition in machine learning of using ensembles to improve model performance. When there is a significant diversity among models, ensembles are expected to yield better results. This ensemble theory is proved to be correct by many ML algorithms; for example,\nAdaBoost\naggregates many weak learners to perform similar or even better than a single strong learner.\nBootstrapping\nensembles multiple trials of resampling to achieve more accurate estimation of metrics. Random forests or\nGBM\nis also a good example for the effectiveness of ensembling.\nTo get better uncertainty estimation, it is intuitive to aggregate a collection of independently trained models. However, it is expensive to train a single deep neural network model, let alone many of them. In reinforcement learning, Bootstrapped DQN (\nOsband, et al. 2016\n) is equipped with multiple value heads and relies on the uncertainty among an ensemble of Q value approximation to guide\nexploration\nin RL.\nIn active learning, a commoner approach is to use\ndropout\nto “simulate” a probabilistic Gaussian process (\nGal & Ghahramani 2016\n). We thus ensemble multiple samples collected from the same model but with different dropout masks applied during the forward pass to estimate the model uncertainty (epistemic uncertainty). The process is named\nMC dropout\n(Monte Carlo dropout), where dropout is applied before every weight layer, is approved to be mathematically equivalent to an approximation to the probabilistic deep Gaussian process (\nGal & Ghahramani 2016\n). This simple idea has been shown to be effective for classification with small datasets and widely adopted in scenarios when efficient model uncertainty estimation is needed.\nDBAL\n(Deep Bayesian active learning;\nGal et al. 2017\n) approximates Bayesian neural networks with MC dropout such that it learns a distribution over model weights. In their experiment, MC dropout performed better than random baseline and mean standard deviation (Mean STD), similarly to variation ratios and entropy measurement.\nActive learning results of DBAL on MNIST. (Image source:\nGal et al. 2017\n).\nBeluch et al. (2018)\ncompared ensemble-based models with MC dropout and found that the combination of naive ensemble (i.e. train multiple models separately and independently) and variation ratio yields better calibrated predictions than others. However, naive ensembles are\nvery\nexpensive, so they explored a few alternative cheaper options:\nSnapshot ensemble: Use a cyclic learning rate schedule to train an implicit ensemble such that it converges to different local minima.\nDiversity encouraging ensemble (DEE): Use a base network trained for a small number of epochs as initialization for $n$ different networks, each trained with dropout to encourage diversity.\nSplit head approach: One base model has multiple heads, each corresponding to one classifier.\nUnfortunately all the cheap implicit ensemble options above perform worse than naive ensembles. Considering the limit on computational resources, MC dropout is still a pretty good and economical choice. Naturally, people also try to combine ensemble and MC dropout (\nPop & Fulop 2018\n) to get a bit of additional performance gain by stochastic ensemble.\nUncertainty in Parameter Space\n#\nBayes-by-backprop\n(\nBlundell et al. 2015\n) measures weight uncertainty in neural networks directly. The method maintains a probability distribution over the weights $\\mathbf{w}$, which is modeled as a variational distribution $q(\\mathbf{w} \\vert \\theta)$ since the true posterior $p(\\mathbf{w} \\vert \\mathcal{D})$ is not tractable directly. The loss is to minimize the KL divergence between $q(\\mathbf{w} \\vert \\theta)$ and $p(\\mathbf{w} \\vert \\mathcal{D})$,\n$$\n\\begin{aligned}\n\\mathcal{L}(\\theta)\n&= \\text{KL}[q(\\mathbf{w}\\vert\\theta) \\| p(\\mathbf{w} \\vert \\mathcal{D})] \\\\ \n&= \\int q(\\mathbf{w}\\vert\\theta) \\log \\frac{q(\\mathbf{w}\\vert\\theta)}{p(\\mathbf{w}) p(\\mathcal{D}\\vert \\mathbf{w})} d\\mathbf{w} \\\\ \n&= \\text{KL}[q(\\mathbf{w}\\vert\\theta) \\| p(w)] - \\mathbb{E}_{q(\\mathbf{w}\\vert\\theta)} [\\log p(\\mathcal{D} \\vert \\mathbf{w})] \\\\\n&\\approx \\log q(\\mathbf{w} \\vert \\theta) - \\log p(\\mathbf{w}) p(\\mathcal{D}\\vert \\mathbf{w}) & \\text{; monte carlo sampling; }q(\\mathbf{w} \\vert \\theta)\\text{ & }p(\\mathbf{w})\\text{ are close.}\n\\end{aligned}\n$$\nThe variational distribution $q$ is typically a Gaussian with diagonal covariance and each weight is sampled from $\\mathcal{N}(\\mu_i, \\sigma_i^2)$. To ensure non-negativity of $\\sigma_i$, it is further parameterized via softplus, $\\sigma_i = \\log(1 + \\exp(\\rho_i))$ where the variational parameters are $\\theta = \\{\\mu_i , \\rho_i\\}^d_{i=1}$.\nThe process of Bayes-by-backprop can be summarized as:\nSample $\\epsilon \\sim \\mathcal{N}(0, I)$\nLet $\\mathbf{w} = \\mu + \\log(1+ \\exp(\\rho)) \\circ \\epsilon$\nLet $\\theta = (\\mu, \\rho)$\nLet $f(\\mathbf{w}, \\theta) = \\log q(\\mathbf{w} \\vert \\theta) - \\log p(\\mathbf{w})p(\\mathcal{D}\\vert \\mathbf{w})$\nCalculate the gradient of $f(\\mathbf{w}, \\theta)$ w.r.t. to $\\mu$ and $\\rho$ and then update $\\theta$.\nUncertainty is measured by sampling different model weights during inference.\nLoss Prediction\n#\nThe loss objective guides model training. A low loss value indicates that a model can make good and accurate predictions.\nYoo & Kweon (2019)\ndesigned a\nloss prediction module\nto predict the loss value for unlabeled inputs, as an estimation of how good a model prediction is on the given data. Data samples are selected if the loss prediction module makes uncertain predictions (high loss value) for them. The loss prediction module is a simple MLP with dropout, that takes several intermediate layer features as inputs and concatenates them after a global average pooling.\nUse the model with a loss prediction module to do active learning selection. (Image source:\nYoo & Kweon 2019\n)\nLet $\\hat{l}$ be the output of the loss prediction module and $l$ be the true loss. When training the loss prediction module, a simple MSE loss $=(l - \\hat{l})^2$ is not a good choice, because the loss decreases in time as the model learns to behave better. A good learning objective should be independent of the scale changes of the target loss. They instead rely on the comparison of sample pairs. Within each batch of size $b$, there are $b/2$ pairs of samples $(\\mathbf{x}_i, \\mathbf{x}_j)$ and the loss prediction model is expected to correctly predict which sample has a larger loss.\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{loss}(\\mathbf{x}_i, \\mathbf{x}_j) &= \\max\\big( 0, -\\mathbb{1}(l(\\mathbf{x}_i), l(\\mathbf{x}_j)) \\cdot (\\hat{l}(\\mathbf{x}_i) - \\hat{l}(\\mathbf{x}_j)) + \\epsilon \\big) \\\\ \n\\text{where } \\mathbb{1}(l_i, l_j) &= \\begin{cases} +1 & \\text{if }l_i > l_j \\\\ -1 & \\text{otherwise} \\end{cases} \n\\end{aligned}\n$$\nwhere $\\epsilon$ is a predefined positive margin constant.\nIn experiments on three vision tasks, active learning selection based on the loss prediction performs better than random baseline, entropy based acquisition and\ncore-set\n.\nActive learning results of loss prediction module based selection, in comparison with other approaches. (Image source:\nYoo & Kweon 2019\n)\nAdversarial Setup\n#\nSinha et al. (2019)\nproposed a GAN-like setup, named\nVAAL\n(Variational Adversarial Active Learning), where a discriminator is trained to distinguish unlabeled data from labeled data. Interestingly, active learning acquisition criteria does not depend on the task performance in VAAL.\nIllustration of VAAL (Variational adversarial active learning). (Image source:\nSinha et al. 2019\n)\nThe $\\beta$-VAE learns a latent feature space $\\mathbf{z}^l \\cup \\mathbf{z}^u$, for labeled and unlabeled data respectively, aiming to\ntrick\nthe discriminator $D(.)$ that all the data points are from the labeled pool;\nThe discriminator $D(.)$ predicts whether a sample is labeled (1) or not (0) based on a latent representation $\\mathbf{z}$. VAAL selects unlabeled samples with low discriminator scores, which indicates that those samples are sufficiently different from previously labeled ones.\nThe loss for VAE representation learning in VAAL contains both a reconstruction part (minimizing the ELBO of given samples) and an adversarial part (labeled and unlabeled data is drawn from the same probability distribution $q_\\phi$):\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{VAE} &= \\lambda_1 \\mathcal{L}^\\text{rec}_\\text{VAE} + \\lambda_2 \\mathcal{L}^\\text{adv}_\\text{VAE} \\\\\n\\mathcal{L}^\\text{rec}_\\text{VAE} &= \\mathbb{E}[\\log p_\\theta(\\mathbf{x}^l \\vert \\mathbf{z}^l)] - \\beta \\text{KL}(q_\\phi(\\mathbf{z}^l \\vert \\mathbf{x}^l) \\| p(\\mathbf{\\tilde{z}})) + \\mathbb{E}[\\log p_\\theta(\\mathbf{u} \\vert \\mathbf{z}^u)] - \\beta \\text{KL}(q_\\phi(\\mathbf{z}^u \\vert \\mathbf{u}) \\| p(\\mathbf{\\tilde{z}})) \\\\\n\\mathcal{L}^\\text{adv}_\\text{VAE} &= - \\mathbb{E}[\\log D(q_\\phi (\\mathbf{z}^l \\vert \\mathbf{x}^l))] - \\mathbb{E}[\\log D(q_\\phi(\\mathbf{z}^u \\vert \\mathbf{u}))]\n\\end{aligned}\n$$\nwhere $p(\\mathbf{\\tilde{z}})$ is a unit Gaussian as a predefined prior and $\\beta$ is the Lagrangian parameter.\nThe discriminator loss is:\n$$\n\\mathcal{L}_D = -\\mathbb{E}[\\log D(q_\\phi (\\mathbf{z}^l \\vert \\mathbf{x}^l))] - \\mathbb{E}[\\log (1 - D(q_\\phi (\\mathbf{z}^u \\vert \\mathbf{u})))]\n$$\nExperiment results of VAAL (variational adversarial active learning) on several image classification tasks. (Image source:\nSinha et al. 2019\nAblation studies showed that jointly training VAE and discriminator is critical. Their results are robust to the biased initial labeled pool, different labeling budgets and noisy oracle.\nMAL\n(Minimax Active Learning;\nEbrahimiet al. 2021\n) is an extension of VAAL. The MAL framework consists of an entropy minimizing feature encoding network $F$ followed by an entropy maximizing classifier $C$. This minimax setup reduces the distribution gap between labeled and unlabeled data.\nIllustration of the MAL (minimax active learning) framework. (Image source:\nEbrahimiet al. 2021\n)\nA feature encoder $F$ encodes a sample into a $\\ell_2$-normalized $d$-dimensional latent vector. Assuming there are $K$ classes, a classifier $C$ is parameterized by $\\mathbf{W} \\in \\mathbb{R}^{d \\times K}$.\n(1) First $F$ and $C$ are trained on labeled samples by a simple cross entropy loss to achieve good classification results,\n$$\n\\mathcal{L}_\\text{CE} = -\\mathbb{E}_{(\\mathbf{x}^l, y) \\sim \\mathcal{X}} \\sum_{k=1}^K \\mathbb{1}[k=y] \\log\\Big( \\sigma(\\frac{1}{T} \\frac{\\mathbf{W}^\\top F\\big(\\mathbf{x}^l)}{\\|F(\\mathbf{x}^l)\\|}\\big) \\Big)\n$$\n(2) When training on the unlabeled examples, MAL relies on a\nminimax\ngame setup\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{Ent} &= -\\sum^K_{k=1} p(y=k \\vert \\mathbf{u}) \\log p(y=k\\vert \\mathbf{u}) \\\\\n\\theta^*_F, \\theta^*_C &= \\min_F\\max_C \\mathcal{L}_\\text{Ent} \\\\\n\\theta_F &\\gets \\theta_F - \\alpha_1 \\nabla \\mathcal{L}_\\text{Ent} \\\\\n\\theta_C &\\gets \\theta_C + \\alpha_2 \\nabla \\mathcal{L}_\\text{Ent}\n\\end{aligned}\n$$\nwhere,\nFirst, minimizing the entropy in $F$ encourages unlabeled samples associated with similar predicted labels to have similar features.\nMaximizing the entropy in $C$ adversarially makes the prediction to follow a more uniform class distribution.\n(My understanding here is that because the true label of an unlabeled sample is unknown, we should not optimize the classifier to maximize the predicted labels just yet.)\nThe discriminator is trained in the same way as in VAAL.\nSampling strategy in MAL considers both diversity and uncertainty:\nDiversity: the score of $D$ indicates how similar a sample is to previously seen examples. A score closer to 0 is better to select unfamiliar data points.\nUncertainty: use the entropy obtained by $C$. A higher entropy score indicates that the model cannot make a confident prediction yet.\nThe experiments compared MAL to random, entropy, core-set, BALD and VAAL baselines, on image classification and segmentation tasks. The results look pretty strong.\nPerformance of MAL on ImageNet. (Table source:\nEbrahimiet al. 2021\n)\nCAL\n(Contrastive Active Learning;\nMargatina et al. 2021\n) intends to select\ncontrastive\nexamples. If two data points with different labels share similar network representations $\\Phi(.)$, they are considered as contrastive examples in CAL. Given a pair of contrastive examples $(\\mathbf{x}_i, \\mathbf{x}_j)$, they should\n$$\nd(\\Phi(\\mathbf{x}_i), \\Phi(\\mathbf{x}_j)) < \\epsilon \\quad\\text{and}\\quad \\text{KL}(p(y\\vert \\mathbf{x}_i) \\| p(y\\vert \\mathbf{x}_j)) \\rightarrow \\infty\n$$\nGiven an unlabeled sample $\\mathbf{x}$, CAL runs the following process:\nSelect the top $k$ nearest neighbors in the model feature space among the labeled samples, $\\{(\\mathbf{x}^l_i, y_i\\}_{i=1}^M \\subset \\mathcal{X}$.\nCompute the KL divergence between the model output probabilities of $\\mathbf{x}$ and each in $\\{\\mathbf{x}^l\\}$. The contrastive score of $\\mathbf{x}$ is the average of these KL divergence values: $s(\\mathbf{x}) = \\frac{1}{M} \\sum_{i=1}^M \\text{KL}(p(y \\vert \\mathbf{x}^l_i | p(y \\vert \\mathbf{x}))$.\nSamples with\nhigh contrastive scores\nare selected for active learning.\nOn a variety of classification tasks, the experiment results of CAL look similar to the entropy baseline.\nMeasuring Representativeness\n#\nCore-sets Approach\n#\nA\ncore-set\nis a concept in computational geometry, referring to a small set of points that approximates the shape of a larger point set. Approximation can be captured by some geometric measure. In the active learning, we expect a model that is trained over the core-set to behave comparably with the model on the entire data points.\nSener & Savarese (2018)\ntreats active learning as a core-set selection problem. Let’s say, there are $N$ samples in total accessible during training. During active learning, a small set of data points get labeled at every time step $t$, denoted as $\\mathcal{S}^{(t)}$. The upper bound of the learning objective can be written as follows, where the\ncore-set loss\nis defined as the difference between average empirical loss over the labeled samples and the loss over the entire dataset including unlabelled ones.\n$$\n\\begin{aligned}\n\\mathbb{E}_{(\\mathbf{x}, y) \\sim p} [\\mathcal{L}(\\mathbf{x}, y)]\n\\leq& \\bigg\\vert \\mathbb{E}_{(\\mathbf{x}, y) \\sim p} [\\mathcal{L}(\\mathbf{x}, y)] - \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}(\\mathbf{x}_i, y_i) \\bigg\\vert & \\text{; Generalization error}\\\\\n+& \\frac{1}{\\vert \\mathcal{S}^{(t)} \\vert} \\sum_{j=1}^{\\vert \\mathcal{S}^{(t)} \\vert} \\mathcal{L}(\\mathbf{x}^l_j, y_j) & \\text{; Training error}\\\\\n+& \\bigg\\vert \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}(\\mathbf{x}_i, y_i) - \\frac{1}{\\vert \\mathcal{S}^{(t)} \\vert} \\sum_{j=1}^{\\vert \\mathcal{S}^{(t)} \\vert} \\mathcal{L}(\\mathbf{x}^l_j, y_j) \\bigg\\vert & \\text{; Core-set error}\n\\end{aligned}\n$$\nThen the active learning problem can be redefined as:\n$$\n\\min_{\\mathcal{S}^{(t+1)} : \\vert \\mathcal{S}^{(t+1)} \\vert \\leq b} \\bigg\\vert \\frac{1}{N}\\sum_{i=1}^N \\mathcal{L}(\\mathbf{x}_i, y_i) - \\frac{1}{\\vert \\mathcal{S}^{(t)} \\cup \\mathcal{S}^{(t+1)} \\vert} \\sum_{j=1}^{\\vert \\mathcal{S}^{(t)} \\cup \\mathcal{S}^{(t+1)} \\vert} \\mathcal{L}(\\mathbf{x}^l_j, y_j) \\bigg\\vert\n$$\nIt is equivalent to\nthe $k$-Center problem\n: choose $b$ center points such that the largest distance between a data point and its nearest center is minimized. This problem is NP-hard. An approximate solution depends on the greedy algorithm.\nActive learning results of core-sets algorithm in comparison with several common baselines on CIFAR-10, CIFAR-100, SVHN. (Image source:\nSener & Savarese 2018\n)\nIt works well on image classification tasks when there is a small number of classes. When the number of classes grows to be large or the data dimensionality increases (“curse of dimensionality”), the core-set method becomes less effective (\nSinha et al. 2019\n).\nBecause the core-set selection is expensive,\nColeman et al. (2020)\nexperimented with a weaker model (e.g. smaller, weaker architecture, not fully trained) and found that empirically using a weaker model as a proxy can significantly shorten each repeated data selection cycle of training models and selecting samples, without hurting the final error much. Their method is referred to as\nSVP\n(Selection via Proxy).\nDiverse Gradient Embedding\n#\nBADGE\n(Batch Active learning by Diverse Gradient Embeddings;\nAsh et al. 2020\n) tracks both model uncertainty and data diversity in the gradient space. Uncertainty is measured by the gradient magnitude w.r.t. the final layer of the network and diversity is captured by a diverse set of samples that span in the gradient space.\nUncertainty. Given an unlabeled sample $\\mathbf{x}$, BADGE first computes the prediction $\\hat{y}$ and the gradient $g_\\mathbf{x}$ of the loss on $(\\mathbf{x}, \\hat{y})$ w.r.t. the last layer’s parameters. They observed that the norm of $g_\\mathbf{x}$ conservatively estimates the example’s influence on the model learning and high-confidence samples tend to have gradient embeddings of small magnitude.\nDiversity. Given many gradient embeddings of many samples, $g_\\mathbf{x}$, BADGE runs\n$k$-means++\nto sample data points accordingly.\nAlgorithm of BADGE (batch active learning by diverse gradient embeddings). (Image source:\nAsh et al. 2020\n)\nMeasuring Training Effects\n#\nQuantify Model Changes\n#\nSettles et al. (2008)\nintroduced an active learning query strategy, named\nEGL\n(Expected Gradient Length). The motivation is to find samples that can trigger the greatest update on the model if their labels are known.\nLet $\\nabla \\mathcal{L}(\\theta)$ be the gradient of the loss function with respect to the model parameters. Specifically, given an unlabeled sample $\\mathbf{x}_i$, we need to calculate the gradient assuming the label is $y \\in \\mathcal{Y}$, $\\nabla \\mathcal{L}^{(y)}(\\theta)$. Because the true label $y_i$ is unknown, EGL relies on the current model belief to compute the expected gradient change:\n$$\n\\text{EGL}(\\mathbf{x}_i) = \\sum_{y_i \\in \\mathcal{Y}} p(y=y_i \\vert \\mathbf{x}) \\|\\nabla \\mathcal{L}^{(y_i)}(\\theta)\\|\n$$\nBALD\n(Bayesian Active Learning by Disagreement;\nHoulsby et al. 2011\n) aims to identify samples to maximize the information gain about the model weights, that is equivalent to maximize the decrease in expected posterior entropy.\n$$\n\\begin{aligned}\nI[\\boldsymbol{\\theta}, y \\vert x,\\mathcal{D}] \n&= H(\\boldsymbol{\\theta} \\vert \\mathcal{D}) - \\mathbb{E}_{y \\sim p(y \\vert \\boldsymbol{x}, \\mathcal{D})} \\big[ H(\\boldsymbol{\\theta} \\vert y, \\boldsymbol{x}, \\mathcal{D}) \\big] & \\text{; Decrease in expected posterior entropy}\\\\ \n&= H(y \\vert \\boldsymbol{x}, \\mathcal{D}) - \\mathbb{E}_{\\boldsymbol{\\theta} \\sim p(\\boldsymbol{\\theta} \\vert \\mathcal{D})} \\big[ H(y \\vert \\boldsymbol{x}, \\mathcal{\\theta}) \\big]\n\\end{aligned}\n$$\nThe underlying interpretation is to “seek $\\mathbf{x}$ for which the model is marginally most uncertain about $y$ (high $H(y \\vert \\mathbf{x}, \\mathcal{D})$), but for which individual settings of the parameters are confident (low $H(y \\vert \\mathbf{x}, \\boldsymbol{\\theta})$).” In other words, each individual posterior draw is confident but a collection of draws carry diverse opinions.\nBALD was originally proposed for an individual sample and\nKirsch et al. (2019)\nextended it to work in batch mode.\nForgetting Events\n#\nTo investigate whether neural networks have a tendency to\nforget\npreviously learned information,\nMariya Toneva et al. (2019)\ndesigned an experiment: They track the model prediction for each sample during the training process and count the transitions for each sample from being classified correctly to incorrectly or vice-versa. Then samples can be categorized accordingly,\nForgettable\n(redundant) samples: If the class label changes across training epochs.\nUnforgettable\nsamples: If the class label assignment is consistent across training epochs. Those samples are never forgotten once learned.\nThey found that there are a large number of unforgettable examples that are never forgotten once learnt. Examples with noisy labels or images with “uncommon” features (visually complicated to classify) are among the most forgotten examples. The experiments empirically validated that unforgettable examples can be safely removed without compromising model performance.\nIn the implementation, the forgetting event is only counted when a sample is included in the current training batch; that is, they compute forgetting across presentations of the same example in subsequent mini-batches. The number of forgetting events per sample is quite stable across different seeds and forgettable examples have a small tendency to be first-time learned later in the training. The forgetting events are also found to be transferable throughout the training period and between architectures.\nForgetting events can be used as a signal for active learning acquisition if we hypothesize a model changing predictions during training is an indicator of model uncertainty. However, ground truth is unknown for unlabeled samples.\nBengar et al. (2021)\nproposed a new metric called\nlabel dispersion\nfor such a purpose. Let’s see across the training time, $c^*$ is the most commonly predicted label for the input $\\mathbf{x}$ and the label dispersion measures the fraction of training steps when the model does not assign $c^**$ to this sample:\n$$\n\\text{Dispersion}(\\mathbf{x}) = 1 - \\frac{f_\\mathbf{x}}{T} \\text{ where }\nf_\\mathbf{x} = \\sum_{t=1}^T \\mathbb{1}[\\hat{y}_t = c^*], c^* = \\arg\\max_{c=1,\\dots,C}\\sum_{t=1}^T \\mathbb{1}[\\hat{y}_t = c]\n$$\nIn their implementation, dispersion is computed at every epoch. Label dispersion is low if the model consistently assigns the same label to the same sample but high if the prediction changes often. Label dispersion is correlated with network uncertainty, as shown in\nLabel dispersion is correlated with network uncertainty. On the x-axis, data points are sorted by label dispersion scores. The y-axis is the model prediction accuracy when the model trys to infer the labels for those samples. (Image source:\nBengar et al. 2021\n)\nHybrid\n#\nWhen running active learning in batch mode, it is important to control diversity within a batch.\nSuggestive Annotation\n(\nSA\n;\nYang et al. 2017\n) is a two-step hybrid strategy, aiming to select both high uncertainty & highly representative labeled samples. It uses uncertainty obtained from an ensemble of models trained on the labeled data and core-sets for choosing representative data samples.\nFirst, SA selects top $K$ images with high uncertainty scores to form a candidate pool $\\mathcal{S}_c \\subseteq \\mathcal{S}_U$. The uncertainty is measured as disagreement between multiple models training with bootstrapping.\nThe next step is to find a subset $\\mathcal{S}_a \\subseteq \\mathcal{S}_c$ with highest representativeness. The cosine similarity between feature vectors of two inputs approximates how similar they are. The representativeness of $\\mathcal{S}_a$ for $\\mathcal{S}_U$ reflects how well $\\mathcal{S}_a$ can represent all the samples in $\\mathcal{S}_u$, defined as:\n$$\nF(\\mathcal{S}_a, \\mathcal{S}_u) = \\sum_{\\mathbf{x}_j \\in \\mathcal{S}_u} f(\\mathcal{S}_a, \\mathbf{x}_j) = \\sum_{\\mathbf{x}_j \\in \\mathcal{S}_u} \\max_{\\mathbf{x}_i \\in \\mathcal{S}_a} \\text{sim}(\\mathbf{x}_i, \\mathbf{x}_j)\n$$\nFormulating $\\mathcal{S}_a \\subseteq \\mathcal{S}_c$ with $k$ data points that maximizes $F(\\mathcal{S}_a, \\mathcal{S}_u)$ is a generalized version of the maximum set cover problem. It is NP-hard and its best possible polynomial time approximation algorithm is a simple greedy method.\nInitially, $\\mathcal{S}_a = \\emptyset$ and $F(\\mathcal{S}_a, \\mathcal{S}_u) = 0$.\nThen, iteratively add $\\mathbf{x}_i \\in \\mathcal{S}_c$ that maximizes $F(\\mathcal{S}_a \\cup I_i, \\mathcal{S}_u)$ over $\\mathcal{S}_a$, until $\\mathcal{S}_s$ contains $k$ images.\nZhdanov (2019)\nruns a similar process as SA, but at step 2, it relies on $k$-means instead of core-set, where the size of the candidate pool is configured relative to the batch size. Given batch size $b$ and a constant $beta$ (between 10 and 50), it follows these steps:\nTrain a classifier on the labeled data;\nMeasure informativeness of every unlabeled example (e.g. using uncertainty metrics);\nPrefilter top $\\beta b \\geq b$ most informative examples;\nCluster $\\beta b$ examples into $B$ clusters;\nSelect $b$ different examples closest to the cluster centers for this round of active learning.\nActive learning can be further combined with\nsemi-supervised learning\nto save the budget.\nCEAL\n(Cost-Effective Active Learning;\nYang et al. 2017\n) runs two things in parallel:\nSelect uncertain samples via active learning and get them labeled;\nSelect samples with the most confident prediction and assign them\npseudo labels\n. The confidence prediction is judged by whether the prediction entropy is below a threshold $\\delta$. As the model is getting better in time, the threshold $\\delta$ decays in time as well.\nIllustration of CEAL (cost-effective active learning). (Image source:\nYang et al. 2017\n)\nCitation\n#\nCited as:\nWeng, Lilian. (Feb 2022). Learning with not enough data part 2: active learning. Lil’Log. https://lilianweng.github.io/posts/2022-02-20-active-learning/.\nOr\n@article{weng2022active,\n title = \"Learning with not Enough Data Part 2: Active Learning\",\n author = \"Weng, Lilian\",\n journal = \"lilianweng.github.io\",\n year = \"2022\",\n month = \"Feb\",\n url = \"https://lilianweng.github.io/posts/2022-02-20-active-learning/\"\n}\nReferences\n#\n[1] Burr Settles.\nActive learning literature survey.\nUniversity of Wisconsin, Madison, 52(55-66):11, 2010.\n[2]\nhttps://jacobgil.github.io/deeplearning/activelearning\n[3] Yang et al.\n“Cost-effective active learning for deep image classification”\nTCSVT 2016.\n[4] Yarin Gal et al.\n“Dropout as a Bayesian Approximation: representing model uncertainty in deep learning.”\nICML 2016.\n[5] Blundell et al.\n“Weight uncertainty in neural networks (Bayes-by-Backprop)”\nICML 2015.\n[6] Settles et al.\n“Multiple-Instance Active Learning.”\nNIPS 2007.\n[7] Houlsby et al.\nBayesian Active Learning for Classification and Preference Learning.\"\narXiv preprint arXiv:1112.5745 (2020).\n[8] Kirsch et al.\n“BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning.”\nNeurIPS 2019.\n[9] Beluch et al.\n“The power of ensembles for active learning in image classification.”\nCVPR 2018.\n[10] Sener & Savarese.\n“Active learning for convolutional neural networks: A core-set approach.”\nICLR 2018.\n[11] Donggeun Yoo & In So Kweon.\n“Learning Loss for Active Learning.”\nCVPR 2019.\n[12] Margatina et al.\n“Active Learning by Acquiring Contrastive Examples.”\nEMNLP 2021.\n[13] Sinha et al.\n“Variational Adversarial Active Learning”\nICCV 2019\n[14] Ebrahimiet al.\n“Minmax Active Learning”\narXiv preprint arXiv:2012.10467 (2021).\n[15] Mariya Toneva et al.\n“An empirical study of example forgetting during deep neural network learning.”\nICLR 2019.\n[16] Javad Zolfaghari Bengar et al.\n“When Deep Learners Change Their Mind: Learning Dynamics for Active Learning.”\nCAIP 2021.\n[17] Yang et al.\n“Suggestive annotation: A deep active learning framework for biomedical image segmentation.”\nMICCAI 2017.\n[18] Fedor Zhdanov.\n“Diverse mini-batch Active Learning”\narXiv preprint arXiv:1901.05954 (2019).", "metadata": { "title": "Learning with not Enough Data Part 2: Active Learning | Lil'Log", "description": "This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.", "keywords": "data, active-learning", "author": "Lilian Weng", "og:title": "Learning with not Enough Data Part 2: Active Learning", "og:description": "This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.", "og:type": "article", "og:url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/", "twitter:card": "summary", "twitter:title": "Learning with not Enough Data Part 2: Active Learning", "twitter:description": "This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.", "article:section": "posts", "article:published_time": "2022-02-20T00:00:00+00:00", "article:modified_time": "2022-02-20T00:00:00+00:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/028a8389cdec/code/body-000.txt", "chars": 276, "language": "text" } ], "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-workflow.png", "bytes": 251848, "reused": false, "local_path": "attachments/028a8389cdec/images/body-d81486b940074ca7bee2.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-workflow.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/DBAL-exp.png", "bytes": 255226, "reused": false, "local_path": "attachments/028a8389cdec/images/body-a6d3abf0912005132083.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/DBAL-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-loss-prediction.png", "bytes": 208573, "reused": false, "local_path": "attachments/028a8389cdec/images/body-533dde5b3386ec869780.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-loss-prediction.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-loss-prediction-exp.png", "bytes": 582217, "reused": false, "local_path": "attachments/028a8389cdec/images/body-a27ab134b2539e4fa927.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-loss-prediction-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/VAAL.png", "bytes": 545888, "reused": false, "local_path": "attachments/028a8389cdec/images/body-f2a382c35839b44510af.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/VAAL.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/VAAL-exp.png", "bytes": 705319, "reused": false, "local_path": "attachments/028a8389cdec/images/body-5b0a42592fcac6b2269f.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/VAAL-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/MAL.png", "bytes": 412179, "reused": false, "local_path": "attachments/028a8389cdec/images/body-f70d56c397cd0359db6b.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/MAL.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/MAL-exp.png", "bytes": 264815, "reused": false, "local_path": "attachments/028a8389cdec/images/body-e108c7cf704e84720825.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/MAL-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/core-sets-exp.png", "bytes": 1047500, "reused": false, "local_path": "attachments/028a8389cdec/images/body-bd8c66b6e5605f2b18ac.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/core-sets-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/BADGE-algo.png", "bytes": 246545, "reused": false, "local_path": "attachments/028a8389cdec/images/body-57159f6aa6eb14ac2a40.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/BADGE-algo.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/label-dispersion-vs-uncertainty.png", "bytes": 217492, "reused": false, "local_path": "attachments/028a8389cdec/images/body-cad8d44f6e9a7e9a6aef.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/label-dispersion-vs-uncertainty.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/CEAL.png", "bytes": 896102, "reused": false, "local_path": "attachments/028a8389cdec/images/body-5b32e280369a4496da10.png", "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/CEAL.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://burrsettles.com/pub/settles.activelearning.pdf", "label": "survey paper", "type": "a" }, { "href": "https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.455.9057&rep=rep1&type=pdf", "label": "Der Kiureghian & Ditlevsen 2009", "type": "a" }, { "href": "https://arxiv.org/abs/1703.04977", "label": "Kendall & Gal 2017", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/AdaBoost", "label": "AdaBoost", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Bootstrapping_(statistics)", "label": "Bootstrapping", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Gradient_boosting", "label": "GBM", "type": "a" }, { "href": "https://arxiv.org/abs/1602.04621", "label": "Osband, et al. 2016", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2020-06-07-exploration-drl/", "label": "exploration", "type": "a" }, { "href": "https://arxiv.org/abs/1506.02142", "label": "Gal & Ghahramani 2016", "type": "a" }, { "href": "https://arxiv.org/abs/1506.02157", "label": "Gal & Ghahramani 2016", "type": "a" }, { "href": "https://arxiv.org/abs/1703.02910", "label": "Gal et al. 2017", "type": "a" }, { "href": "https://openaccess.thecvf.com/content_cvpr_2018/papers/Beluch_The_Power_of_CVPR_2018_paper.pdf", "label": "Beluch et al. (2018)", "type": "a" }, { "href": "https://arxiv.org/abs/1811.03897", "label": "Pop & Fulop 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1505.05424", "label": "Blundell et al. 2015", "type": "a" }, { "href": "https://arxiv.org/abs/1905.03677", "label": "Yoo & Kweon (2019)", "type": "a" }, { "href": "https://arxiv.org/abs/1904.00370", "label": "Sinha et al. (2019)", "type": "a" }, { "href": "https://arxiv.org/abs/2012.10467", "label": "Ebrahimiet al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2109.03764", "label": "Margatina et al. 2021", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2021-05-31-contrastive/", "label": "contrastive", "type": "a" }, { "href": "https://arxiv.org/abs/1708.00489", "label": "Sener & Savarese (2018)", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Metric_k-center", "label": "the $k$-Center problem", "type": "a" }, { "href": "https://arxiv.org/abs/1906.11829", "label": "Coleman et al. (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/1906.03671", "label": "Ash et al. 2020", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/K-means%2B%2B", "label": "$k$-means++", "type": "a" }, { "href": "https://papers.nips.cc/paper/2007/hash/a1519de5b5d44b31a01de013b9b51a80-Abstract.html", "label": "Settles et al. (2008)", "type": "a" }, { "href": "https://arxiv.org/abs/1112.5745", "label": "Houlsby et al. 2011", "type": "a" }, { "href": "https://arxiv.org/abs/1906.08158", "label": "Kirsch et al. (2019)", "type": "a" }, { "href": "https://arxiv.org/abs/1812.05159", "label": "Mariya Toneva et al. (2019)", "type": "a" }, { "href": "https://arxiv.org/abs/2107.14707", "label": "Bengar et al. (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/1706.04737", "label": "Yang et al. 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1901.05954", "label": "Zhdanov (2019)", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/", "label": "semi-supervised learning", "type": "a" }, { "href": "https://arxiv.org/abs/1701.03551", "label": "Yang et al. 2017", "type": "a" }, { "href": "https://jacobgil.github.io/deeplearning/activelearning", "label": "https://jacobgil.github.io/deeplearning/activelearning", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:51:24Z", "source_url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/", "papers": [ { "status": "ok", "url": "https://burrsettles.com/pub/settles.activelearning.pdf", "bytes": 1939936, "reused": false, "local_path": "papers/repair/1a9a9c6742fab449158e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1703.04977", "bytes": 6719323, "reused": false, "local_path": "papers/repair/266ae612868b87b732a4.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1602.04621", "bytes": 6874097, "reused": false, "local_path": "papers/repair/d7a1d129f52a82d595ff.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1506.02142", "bytes": 1107246, "reused": false, "local_path": "papers/repair/4fb011fb3475ef624220.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1506.02157", "bytes": 436231, "reused": false, "local_path": "papers/repair/f341fa9dd0f0f593ee60.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1703.02910", "bytes": 724240, "reused": false, "local_path": "papers/repair/523334b8d44dca5de9f0.pdf" }, { "status": "ok", "url": "https://openaccess.thecvf.com/content_cvpr_2018/papers/Beluch_The_Power_of_CVPR_2018_paper.pdf", "bytes": 1715354, "reused": false, "local_path": "papers/repair/f0bcca0541114ac25b32.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1811.03897", "bytes": 4205547, "reused": false, "local_path": "papers/repair/ce97c4ced1411688db16.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1505.05424", "bytes": 628773, "reused": false, "local_path": "papers/repair/475ff08ad5512624a7cc.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.03677", "bytes": 564131, "reused": false, "local_path": "papers/repair/79dc6ada83281404ff9d.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1904.00370", "bytes": 930575, "reused": false, "local_path": "papers/repair/79c5aedc49e45da7dda8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2012.10467", "bytes": 1339851, "reused": false, "local_path": "papers/repair/90cd58e430cea1408452.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2109.03764", "bytes": 2036447, "reused": false, "local_path": "papers/repair/f5ba7910af8e3a2395ac.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1708.00489", "bytes": 5033708, "reused": false, "local_path": "papers/repair/238fc8d82668ad254fc1.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1906.11829", "bytes": 17351339, "reused": false, "local_path": "papers/repair/ad0851b7cc881c207705.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1906.03671", "bytes": 4265486, "reused": false, "local_path": "papers/repair/7474699a4e844a465ace.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1112.5745", "bytes": 556354, "reused": false, "local_path": "papers/repair/d0182f4c21330f972a26.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1906.08158", "bytes": 4001227, "reused": false, "local_path": "papers/repair/b525dbbc20e226a0e95a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1812.05159", "bytes": 1731989, "reused": true, "local_path": "papers/repair/55cf5f36e50e938900cd.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2107.14707", "bytes": 488570, "reused": false, "local_path": "papers/repair/27448d46f2a1e682c257.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1706.04737", "bytes": 5679851, "reused": false, "local_path": "papers/repair/de814b520904eeb1ed6e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1901.05954", "bytes": 387045, "reused": false, "local_path": "papers/repair/3bd458965ad0d9c6d253.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1701.03551", "bytes": 2102878, "reused": false, "local_path": "papers/repair/85a5a6a9cc2ce15af84e.pdf" } ], "actual_video_files": [], "body_chars": 32971, "expected_body_images": 12, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/028a8389cdec/pages/source.html", "local_html": "attachments/028a8389cdec/pages/article.html", "body_base_path": "attachments/028a8389cdec/pages/", "body_markdown_path": "attachments/028a8389cdec/pages/article.md" } }, { "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/", "status": "ok", "article_id": "c8f3f2a59bb8", "local_path": "series-follow/1602/repair/0322612d07df5f51284a.html", "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/triplet-loss.png", "bytes": 82669, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/c8feb8e3831d6f407222.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/lifted-structured-loss.png", "bytes": 238558, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/864616500d13b290a57d.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-sampling-bias.png", "bytes": 618000, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/4d1eafbd2698e4b13414.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-debias-t-SNE.png", "bytes": 447800, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/2202dbf059680ff25f64.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-hard-negatives-code.png", "bytes": 331316, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/56fdc531021fd8fc266d.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCLR.png", "bytes": 92328, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/a3de29261702db65276c.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCLR-algo.png", "bytes": 292083, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/9fbb7be242e78cf4064e.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/barlow-twins.png", "bytes": 165448, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/7c6e570b2b4acaea53ee.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/barlow-twins-algo.png", "bytes": 410905, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/0dc7fb2e267a076918a0.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/BYOL.png", "bytes": 160971, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/64c06b583b29bc8d849d.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/instance-level-discrimination.png", "bytes": 575598, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/24abca83dac665bdfd2c.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/MoCo.png", "bytes": 191327, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/68a57a589ced1a8dcbeb.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CURL.png", "bytes": 458778, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/dec760a7d6d875b1e5f6.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/deepcluster.png", "bytes": 413067, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/3b05f2c9b09900dd678c.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SwAV.png", "bytes": 307781, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/9fe78703c0d6dd59901b.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP.png", "bytes": 395407, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/bcffab9e56e403b12eb6.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP-algo.png", "bytes": 378369, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/14fc3bb2a028bd633430.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP-efficiency.png", "bytes": 269518, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/1cdb06387d4aabe19e7f.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/sup-con.png", "bytes": 1172082, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/dc3294612801cee60ee0.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/EDA-exp1.png", "bytes": 299081, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/4ecf4ae1a9c6c75e0b98.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/text-cutoff.png", "bytes": 102477, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/2f938fd27f3112a7668b.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCSE.png", "bytes": 302804, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/476b14449fa38214486c.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCSE-STS-exp.png", "bytes": 459618, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/d417f179aa932198443f.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SBERT.png", "bytes": 248555, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/61290a159906f77df170.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SBERT-SentEval.png", "bytes": 263986, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/25636cdf26fe48bc32a6.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/BERT-flow.png", "bytes": 127994, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/1f6149bb39a134da1e75.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/whitening-SBERT.png", "bytes": 167429, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/75a0434d42ea9e13e5f2.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/quick-thought.png", "bytes": 235880, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/0873d959165b59705852.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/IS-BERT.png", "bytes": 403903, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/7fc9d0b8fcf86efd800b.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/IS-BERT-SentEval.png", "bytes": 237372, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/4a85aecfe0dc34b27289.png" } ], "article": { "source_file": "0930", "source_platform": "个人博客", "author": "Lilian Weng / Lil'Log", "title_listed": "Contrastive Representation Learning", "keywords": "对比学习,表示学习,自监督", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/", "publish_date": "2021-05", "corpus_stage": "预训练/SFT", "notes": "对比表示学习综述（SimCLR、MoCo等）", "id": "c8f3f2a59bb8", "collected_at": "2026-10-02T14:12:57", "title": "Contrastive Representation Learning | Lil'Log", "body_markdown": "The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in\nself-supervised learning\n.\nContrastive Training Objectives\n#\nIn early versions of loss functions for contrastive learning, only one positive and one negative sample are involved. The trend in recent training objectives is to include multiple positive and negative pairs in one batch.\nContrastive Loss\n#\nContrastive loss\n(\nChopra et al. 2005\n) is one of the earliest training objectives used for deep metric learning in a contrastive fashion.\nGiven a list of input samples $\\{ \\mathbf{x}_i \\}$, each has a corresponding label $y_i \\in \\{1, \\dots, L\\}$ among $L$ classes. We would like to learn a function $f_\\theta(.): \\mathcal{X}\\to\\mathbb{R}^d$ that encodes $x_i$ into an embedding vector such that examples from the same class have similar embeddings and samples from different classes have very different ones. Thus, contrastive loss takes a pair of inputs $(x_i, x_j)$ and minimizes the embedding distance when they are from the same class but maximizes the distance otherwise.\n$$\n\\mathcal{L}_\\text{cont}(\\mathbf{x}_i, \\mathbf{x}_j, \\theta) = \\mathbb{1}[y_i=y_j] \\| f_\\theta(\\mathbf{x}_i) - f_\\theta(\\mathbf{x}_j) \\|^2_2 + \\mathbb{1}[y_i\\neq y_j]\\max(0, \\epsilon - \\|f_\\theta(\\mathbf{x}_i) - f_\\theta(\\mathbf{x}_j)\\|_2)^2\n$$\nwhere $\\epsilon$ is a hyperparameter, defining the lower bound distance between samples of different classes.\nTriplet Loss\n#\nTriplet loss\nwas originally proposed in the FaceNet (\nSchroff et al. 2015\n) paper and was used to learn face recognition of the same person at different poses and angles.\nIllustration of triplet loss given one positive and one negative per anchor. (Image source:\nSchroff et al. 2015\n)\nGiven one anchor input $\\mathbf{x}$, we select one positive sample $\\mathbf{x}^+$ and one negative $\\mathbf{x}^-$, meaning that $\\mathbf{x}^+$ and $\\mathbf{x}$ belong to the same class and $\\mathbf{x}^-$ is sampled from another different class. Triplet loss learns to minimize the distance between the anchor $\\mathbf{x}$ and positive $\\mathbf{x}^+$ and maximize the distance between the anchor $\\mathbf{x}$ and negative $\\mathbf{x}^-$ at the same time with the following equation:\n$$\n\\mathcal{L}_\\text{triplet}(\\mathbf{x}, \\mathbf{x}^+, \\mathbf{x}^-) = \\sum_{\\mathbf{x} \\in \\mathcal{X}} \\max\\big( 0, \\|f(\\mathbf{x}) - f(\\mathbf{x}^+)\\|^2_2 - \\|f(\\mathbf{x}) - f(\\mathbf{x}^-)\\|^2_2 + \\epsilon \\big)\n$$\nwhere the margin parameter $\\epsilon$ is configured as the minimum offset between distances of similar vs dissimilar pairs.\nIt is crucial to select challenging $\\mathbf{x}^-$ to truly improve the model.\nLifted Structured Loss\n#\nLifted Structured Loss\n(\nSong et al. 2015\n) utilizes all the pairwise edges within one training batch for better computational efficiency.\nIllustration compares contrastive loss, triplet loss and lifted structured loss. Red and blue edges connect similar and dissimilar sample pairs respectively. (Image source:\nSong et al. 2015\n)\nLet $D_{ij} = | f(\\mathbf{x}_i) - f(\\mathbf{x}_j) |_2$, a structured loss function is defined as\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{struct} &= \\frac{1}{2\\vert \\mathcal{P} \\vert} \\sum_{(i,j) \\in \\mathcal{P}} \\max(0, \\mathcal{L}_\\text{struct}^{(ij)})^2 \\\\\n\\text{where } \\mathcal{L}_\\text{struct}^{(ij)} &= D_{ij} + \\color{red}{\\max \\big( \\max_{(i,k)\\in \\mathcal{N}} \\epsilon - D_{ik}, \\max_{(j,l)\\in \\mathcal{N}} \\epsilon - D_{jl} \\big)}\n\\end{aligned}\n$$\nwhere $\\mathcal{P}$ contains the set of positive pairs and $\\mathcal{N}$ is the set of negative pairs. Note that the dense pairwise squared distance matrix can be easily computed per training batch.\nThe\nred\npart in $\\mathcal{L}_\\text{struct}^{(ij)}$ is used for mining hard negatives. However, it is not smooth and may cause the convergence to a bad local optimum in practice. Thus, it is relaxed to be:\n$$\n\\mathcal{L}_\\text{struct}^{(ij)} = D_{ij} + \\log \\Big( \\sum_{(i,k)\\in\\mathcal{N}} \\exp(\\epsilon - D_{ik}) + \\sum_{(j,l)\\in\\mathcal{N}} \\exp(\\epsilon - D_{jl}) \\Big)\n$$\nIn the paper, they also proposed to enhance the quality of negative samples in each batch by actively incorporating difficult negative samples given a few random positive pairs.\nN-pair Loss\n#\nMulti-Class N-pair loss\n(\nSohn 2016\n) generalizes triplet loss to include comparison with multiple negative samples.\nGiven a $(N + 1)$-tuplet of training samples, $\\{ \\mathbf{x}, \\mathbf{x}^+, \\mathbf{x}^-_1, \\dots, \\mathbf{x}^-_{N-1} \\}$, including one positive and $N-1$ negative ones, N-pair loss is defined as:\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{N-pair}(\\mathbf{x}, \\mathbf{x}^+, \\{\\mathbf{x}^-_i\\}^{N-1}_{i=1}) \n&= \\log\\big(1 + \\sum_{i=1}^{N-1} \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^-_i) - f(\\mathbf{x})^\\top f(\\mathbf{x}^+))\\big) \\\\\n&= -\\log\\frac{\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+))}{\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+)) + \\sum_{i=1}^{N-1} \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^-_i))}\n\\end{aligned}\n$$\nIf we only sample one negative sample per class, it is equivalent to the softmax loss for multi-class classification.\nNCE\n#\nNoise Contrastive Estimation\n, short for\nNCE\n, is a method for estimating parameters of a statistical model, proposed by\nGutmann & Hyvarinen\nin 2010. The idea is to run logistic regression to tell apart the target data from noise. Read more on how NCE is used for learning word embedding\nhere\n.\nLet $\\mathbf{x}$ be the target sample $\\sim P(\\mathbf{x} \\vert C=1; \\theta) = p_\\theta(\\mathbf{x})$ and $\\tilde{\\mathbf{x}}$ be the noise sample $\\sim P(\\tilde{\\mathbf{x}} \\vert C=0) = q(\\tilde{\\mathbf{x}})$. Note that the logistic regression models the logit (i.e. log-odds) and in this case we would like to model the logit of a sample $u$ from the target data distribution instead of the noise distribution:\n$$\n\\ell_\\theta(\\mathbf{u}) = \\log \\frac{p_\\theta(\\mathbf{u})}{q(\\mathbf{u})} = \\log p_\\theta(\\mathbf{u}) - \\log q(\\mathbf{u})\n$$\nAfter converting logits into probabilities with sigmoid $\\sigma(.)$, we can apply cross entropy loss:\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{NCE} &= - \\frac{1}{N} \\sum_{i=1}^N \\big[ \\log \\sigma (\\ell_\\theta(\\mathbf{x}_i)) + \\log (1 - \\sigma (\\ell_\\theta(\\tilde{\\mathbf{x}}_i))) \\big] \\\\\n\\text{ where }\\sigma(\\ell) &= \\frac{1}{1 + \\exp(-\\ell)} = \\frac{p_\\theta}{p_\\theta + q}\n\\end{aligned}\n$$\nHere I listed the original form of NCE loss which works with only one positive and one noise sample. In many follow-up works, contrastive loss incorporating multiple negative samples is also broadly referred to as NCE.\nInfoNCE\n#\nThe\nInfoNCE loss\nin CPC (\nContrastive Predictive Coding\n;\nvan den Oord, et al. 2018\n), inspired by\nNCE\n, uses categorical cross-entropy loss to identify the positive sample amongst a set of unrelated noise samples.\nGiven a context vector $\\mathbf{c}$, the positive sample should be drawn from the conditional distribution $p(\\mathbf{x} \\vert \\mathbf{c})$, while $N-1$ negative samples are drawn from the proposal distribution $p(\\mathbf{x})$, independent from the context $\\mathbf{c}$. For brevity, let us label all the samples as $X=\\{ \\mathbf{x}_i \\}^N_{i=1}$ among which only one of them $\\mathbf{x}_\\texttt{pos}$ is a positive sample. The probability of we detecting the positive sample correctly is:\n$$\np(C=\\texttt{pos} \\vert X, \\mathbf{c}) \n= \\frac{p(x_\\texttt{pos} \\vert \\mathbf{c}) \\prod_{i=1,\\dots,N; i \\neq \\texttt{pos}} p(\\mathbf{x}_i)}{\\sum_{j=1}^N \\big[ p(\\mathbf{x}_j \\vert \\mathbf{c}) \\prod_{i=1,\\dots,N; i \\neq j} p(\\mathbf{x}_i) \\big]}\n= \\frac{ \\frac{p(\\mathbf{x}_\\texttt{pos}\\vert c)}{p(\\mathbf{x}_\\texttt{pos})} }{ \\sum_{j=1}^N \\frac{p(\\mathbf{x}_j\\vert \\mathbf{c})}{p(\\mathbf{x}_j)} }\n= \\frac{f(\\mathbf{x}_\\texttt{pos}, \\mathbf{c})}{ \\sum_{j=1}^N f(\\mathbf{x}_j, \\mathbf{c}) }\n$$\nwhere the scoring function is $f(\\mathbf{x}, \\mathbf{c}) \\propto \\frac{p(\\mathbf{x}\\vert\\mathbf{c})}{p(\\mathbf{x})}$.\nThe InfoNCE loss optimizes the negative log probability of classifying the positive sample correctly:\n$$\n\\mathcal{L}_\\text{InfoNCE} = - \\mathbb{E} \\Big[\\log \\frac{f(\\mathbf{x}, \\mathbf{c})}{\\sum_{\\mathbf{x}' \\in X} f(\\mathbf{x}', \\mathbf{c})} \\Big]\n$$\nThe fact that $f(x, c)$ estimates the density ratio $\\frac{p(x\\vert c)}{p(x)}$ has a connection with mutual information optimization. To maximize the the mutual information between input $x$ and context vector $c$, we have:\n$$\nI(\\mathbf{x}; \\mathbf{c}) = \\sum_{\\mathbf{x}, \\mathbf{c}} p(\\mathbf{x}, \\mathbf{c}) \\log\\frac{p(\\mathbf{x}, \\mathbf{c})}{p(\\mathbf{x})p(\\mathbf{c})} = \\sum_{\\mathbf{x}, \\mathbf{c}} p(\\mathbf{x}, \\mathbf{c})\\log\\color{blue}{\\frac{p(\\mathbf{x}|\\mathbf{c})}{p(\\mathbf{x})}}\n$$\nwhere the logarithmic term in\nblue\nis estimated by $f$.\nFor sequence prediction tasks, rather than modeling the future observations $p_k(\\mathbf{x}_{t+k} \\vert \\mathbf{c}_t)$ directly (which could be fairly expensive), CPC models a density function to preserve the mutual information between $\\mathbf{x}_{t+k}$ and $\\mathbf{c}_t$:\n$$\nf_k(\\mathbf{x}_{t+k}, \\mathbf{c}_t) = \\exp(\\mathbf{z}_{t+k}^\\top \\mathbf{W}_k \\mathbf{c}_t) \\propto \\frac{p(\\mathbf{x}_{t+k}\\vert\\mathbf{c}_t)}{p(\\mathbf{x}_{t+k})}\n$$\nwhere $\\mathbf{z}_{t+k}$ is the encoded input and $\\mathbf{W}_k$ is a trainable weight matrix.\nSoft-Nearest Neighbors Loss\n#\nSoft-Nearest Neighbors Loss\n(\nSalakhutdinov & Hinton 2007\n,\nFrosst et al. 2019\n) extends it to include multiple positive samples.\nGiven a batch of samples, $\\{\\mathbf{x}_i, y_i)\\}^B_{i=1}$ where $y_i$ is the class label of $\\mathbf{x}_i$ and a function $f(.,.)$ for measuring similarity between two inputs, the soft nearest neighbor loss at temperature $\\tau$ is defined as:\n$$\n\\mathcal{L}_\\text{snn} = -\\frac{1}{B}\\sum_{i=1}^B \\log \\frac{\\sum_{i\\neq j, y_i = y_j, j=1,\\dots,B} \\exp(- f(\\mathbf{x}_i, \\mathbf{x}_j) / \\tau)}{\\sum_{i\\neq k, k=1,\\dots,B} \\exp(- f(\\mathbf{x}_i, \\mathbf{x}_k) /\\tau)}\n$$\nThe temperature $\\tau$ is used for tuning how concentrated the features are in the representation space. For example, when at low temperature, the loss is dominated by the small distances and widely separated representations cannot contribute much and become irrelevant.\nCommon Setup\n#\nWe can loosen the definition of “classes” and “labels” in soft nearest-neighbor loss to create positive and negative sample pairs out of unsupervised data by, for example, applying data augmentation to create noise versions of original samples.\nMost recent studies follow the following definition of contrastive learning objective to incorporate multiple positive and negative samples. According to the setup in (\nWang & Isola 2020\n), let $p_\\texttt{data}(.)$ be the data distribution over $\\mathbb{R}^n$ and $p_\\texttt{pos}(., .)$ be the distribution of positive pairs over $\\mathbb{R}^{n \\times n}$. These two distributions should satisfy:\nSymmetry: $\\forall \\mathbf{x}, \\mathbf{x}^+, p_\\texttt{pos}(\\mathbf{x}, \\mathbf{x}^+) = p_\\texttt{pos}(\\mathbf{x}^+, \\mathbf{x})$\nMatching marginal: $\\forall \\mathbf{x}, \\int p_\\texttt{pos}(\\mathbf{x}, \\mathbf{x}^+) d\\mathbf{x}^+ = p_\\texttt{data}(\\mathbf{x})$\nTo learn an encoder $f(\\mathbf{x})$ to learn a\nL2-normalized feature vector\n, the contrastive learning objective is:\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{contrastive} \n&= \\mathbb{E}_{(\\mathbf{x},\\mathbf{x}^+)\\sim p_\\texttt{pos}, \\{\\mathbf{x}^-_i\\}^M_{i=1} \\overset{\\text{i.i.d}}{\\sim} p_\\texttt{data} } \\Big[ -\\log\\frac{\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+) / \\tau)}{ \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+) / \\tau) + \\sum_{i=1}^M \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}_i^-) / \\tau)} \\Big] & \\\\\n&\\approx \\mathbb{E}_{(\\mathbf{x},\\mathbf{x}^+)\\sim p_\\texttt{pos}, \\{\\mathbf{x}^-_i\\}^M_{i=1} \\overset{\\text{i.i.d}}{\\sim} p_\\texttt{data} }\\Big[ - f(\\mathbf{x})^\\top f(\\mathbf{x}^+) / \\tau + \\log\\big(\\sum_{i=1}^M \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}_i^-) / \\tau)\\big) \\Big] & \\scriptstyle{\\text{; Assuming infinite negatives}} \\\\\n&= -\\frac{1}{\\tau}\\mathbb{E}_{(\\mathbf{x},\\mathbf{x}^+)\\sim p_\\texttt{pos}}f(\\mathbf{x})^\\top f(\\mathbf{x}^+) + \\mathbb{E}_{ \\mathbf{x} \\sim p_\\texttt{data}} \\Big[ \\log \\mathbb{E}_{\\mathbf{x}^- \\sim p_\\texttt{data}} \\big[ \\sum_{i=1}^M \\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}_i^-) / \\tau)\\big] \\Big] &\n\\end{aligned}\n$$\nKey Ingredients\n#\nHeavy Data Augmentation\n#\nGiven a training sample, data augmentation techniques are needed for creating noise versions of itself to feed into the loss as positive samples. Proper data augmentation setup is critical for learning good and generalizable embedding features. It introduces the non-essential variations into examples without modifying semantic meanings and thus encourages the model to learn the essential part of the representation. For example, experiments in\nSimCLR\nshowed that the composition of random cropping and random color distortion is crucial for good performance on learning visual representation of images.\nLarge Batch Size\n#\nUsing a large batch size during training is another key ingredient in the success of many contrastive learning methods (e.g.\nSimCLR\n,\nCLIP\n), especially when it relies on in-batch negatives. Only when the batch size is big enough, the loss function can cover a diverse enough collection of negative samples, challenging enough for the model to learn meaningful representation to distinguish different examples.\nHard Negative Mining\n#\nHard negative samples should have different labels from the anchor sample, but have embedding features very close to the anchor embedding. With access to ground truth labels in supervised datasets, it is easy to identify task-specific hard negatives. For example when learning sentence embedding, we can treat sentence pairs labelled as “contradiction” in NLI datasets as hard negative pairs (e.g.\nSimCSE\n, or use top incorrect candidates returned by BM25 with most keywords matched as hard negative samples (\nDPR\n;\nKarpukhin et al., 2020\n).\nHowever, it becomes tricky to do hard negative mining when we want to remain unsupervised. Increasing training batch size or\nmemory bank\nsize implicitly introduces more hard negative samples, but it leads to a heavy burden of large memory usage as a side effect.\nChuang et al. (2020)\nstudied the sampling bias in contrastive learning and proposed debiased loss. In the unsupervised setting, since we do not know the ground truth labels, we may accidentally sample false negative samples. Sampling bias can lead to significant performance drop.\nSampling bias which refers to false negative samples in contrastive learning can lead to a big performance drop. (Image source:\nChuang et al., 2020\n)\nLet us assume the probability of anchor class $c$ is uniform $\\rho(c)=\\eta^+$ and the probability of observing a different class is $\\eta^- = 1-\\eta^+$.\nThe probability of observing a positive example for $\\mathbf{x}$ is $p^+_x(\\mathbf{x}’)=p(\\mathbf{x}’\\vert \\mathbf{h}_{x’}=\\mathbf{h}_x)$;\nThe probability of getting a negative sample for $\\mathbf{x}$ is $p^-_x(\\mathbf{x}’)=p(\\mathbf{x}’\\vert \\mathbf{h}_{x’}\\neq\\mathbf{h}_x)$.\nWhen we are sampling $\\mathbf{x}^-$ , we cannot access the true $p^-_x(\\mathbf{x}^-)$ and thus $\\mathbf{x}^-$ may be sampled from the (undesired) anchor class $c$ with probability $\\eta^+$. The actual sampling data distribution becomes:\n$$\np(\\mathbf{x}') = \\eta^+ p^+_x(\\mathbf{x}') + \\eta^- p_x^-(\\mathbf{x}')\n$$\nThus we can use $p^-_x(\\mathbf{x}’) = (p(\\mathbf{x}’) - \\eta^+ p^+_x(\\mathbf{x}’))/\\eta^-$ for sampling $\\mathbf{x}^-$ to debias the loss. With $N$ samples $\\{\\mathbf{u}_i\\}^N_{i=1}$ from $p$ and $M$ samples $\\{ \\mathbf{v}_i \\}_{i=1}^M$ from $p^+_x$ , we can estimate the expectation of the second term $\\mathbb{E}_{\\mathbf{x}^-\\sim p^-_x}[\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^-))]$ in the denominator of contrastive learning loss:\n$$\ng(\\mathbf{x}, \\{\\mathbf{u}_i\\}^N_{i=1}, \\{\\mathbf{v}_i\\}_{i=1}^M) = \\max\\Big\\{ \\frac{1}{\\eta^-}\\Big( \\frac{1}{N}\\sum_{i=1}^N \\exp(f(\\mathbf{x})^\\top f(\\mathbf{u}_i)) - \\frac{\\eta^+}{M}\\sum_{i=1}^M \\exp(f(\\mathbf{x})^\\top f(\\mathbf{v}_i)) \\Big), \\exp(-1/\\tau) \\Big\\}\n$$\nwhere $\\tau$ is the temperature and $\\exp(-1/\\tau)$ is the theoretical lower bound of $\\mathbb{E}_{\\mathbf{x}^-\\sim p^-_x}[\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^-))]$.\nThe final debiased contrastive loss looks like:\n$$\n\\mathcal{L}^{N,M}_\\text{debias}(f) = \\mathbb{E}_{\\mathbf{x},\\{\\mathbf{u}_i\\}^N_{i=1}\\sim p;\\;\\mathbf{x}^+, \\{\\mathbf{v}_i\\}_{i=1}^M\\sim p^+} \\Big[ -\\log\\frac{\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+)}{\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^+) + N g(x,\\{\\mathbf{u}_i\\}^N_{i=1}, \\{\\mathbf{v}_i\\}_{i=1}^M)} \\Big]\n$$\nt-SNE visualization of learned representation with debiased contrastive learning. (Image source:\nChuang et al., 2020\n)\nFollowing the above annotation,\nRobinson et al. (2021)\nmodified the sampling probabilities to target at hard negatives by up-weighting the probability $p^-_x(x’)$ to be proportional to its similarity to the anchor sample. The new sampling probability $q_\\beta(x^-)$ is:\n$$\nq_\\beta(\\mathbf{x}^-) \\propto \\exp(\\beta f(\\mathbf{x})^\\top f(\\mathbf{x}^-)) \\cdot p(\\mathbf{x}^-)\n$$\nwhere $\\beta$ is a hyperparameter to tune.\nWe can estimate the second term in the denominator $\\mathbb{E}_{\\mathbf{x}^- \\sim q_\\beta} [\\exp(f(\\mathbf{x})^\\top f(\\mathbf{x}^-))]$ using importance sampling where both the partition functions $Z_\\beta, Z^+_\\beta$ can be estimated empirically.\n$$\n\\begin{aligned}\n\\mathbb{E}_{\\mathbf{u} \\sim q_\\beta} [\\exp(f(\\mathbf{x})^\\top f(\\mathbf{u}))] &= \\mathbb{E}_{\\mathbf{u} \\sim p} [\\frac{q_\\beta}{p}\\exp(f(\\mathbf{x})^\\top f(\\mathbf{u}))] = \\mathbb{E}_{\\mathbf{u} \\sim p} [\\frac{1}{Z_\\beta}\\exp((\\beta + 1)f(\\mathbf{x})^\\top f(\\mathbf{u}))] \\\\\n\\mathbb{E}_{\\mathbf{v} \\sim q^+_\\beta} [\\exp(f(\\mathbf{x})^\\top f(\\mathbf{v}))] &= \\mathbb{E}_{\\mathbf{v} \\sim p^+} [\\frac{q^+_\\beta}{p}\\exp(f(\\mathbf{x})^\\top f(\\mathbf{v}))] = \\mathbb{E}_{\\mathbf{v} \\sim p} [\\frac{1}{Z^+_\\beta}\\exp((\\beta + 1)f(\\mathbf{x})^\\top f(\\mathbf{v}))]\n\\end{aligned}\n$$\nPseudo code for computing NCE loss, debiased contrastive loss, and hard negative sample objective when setting $M=1$. (Image source:\nRobinson et al., 2021\n)\nVision: Image Embedding\n#\nImage Augmentations\n#\nMost approaches for contrastive representation learning in the vision domain rely on creating a noise version of a sample by applying a sequence of data augmentation techniques. The augmentation should significantly change its visual appearance but keep the semantic meaning unchanged.\nBasic Image Augmentation\n#\nThere are many ways to modify an image while retaining its semantic meaning. We can use any one of the following augmentation or a composition of multiple operations.\nRandom cropping and then resize back to the original size.\nRandom color distortions\nRandom Gaussian blur\nRandom color jittering\nRandom horizontal flip\nRandom grayscale conversion\nMulti-crop augmentation: Use two standard resolution crops and sample a set of additional low resolution crops that cover only small parts of the image. Using low resolution crops reduces the compute cost. (\nSwAV\n)\nAnd many more …\nAugmentation Strategies\n#\nMany frameworks are designed for learning good data augmentation strategies (i.e. a composition of multiple transforms). Here are a few common ones.\nAutoAugment\n(\nCubuk, et al. 2018\n): Inspired by\nNAS\n, AutoAugment frames the problem of learning best data augmentation operations (i.e. shearing, rotation, invert, etc.) for image classification as an RL problem and looks for the combination that leads to the highest accuracy on the evaluation set.\nRandAugment (\nCubuk et al., 2019\n): RandAugment greatly reduces the search space of AutoAugment by controlling the magnitudes of different transformation operations with a single magnitude parameter.\nPBA (Population based augmentation;\nHo et al., 2019\n): PBA combined PBT (\nJaderberg et al, 2017\n) with AutoAugment, using the evolutionary algorithm to train a population of children models in parallel to evolve the best augmentation strategies.\nUDA (Unsupervised Data Augmentation;\nXie et al., 2019\n): Among a set of possible augmentation strategies, UDA selects those to minimize the KL divergence between the predicted distribution over an unlabelled example and its unlabelled augmented version.\nImage Mixture\n#\nImage mixture methods can construct new training examples from existing data points.\nMixup (\nZhang et al., 2018\n): It runs global-level mixture by creating a weighted pixel-wise combination of two existing images $I_1$ and $I_2$: $I_\\text{mixup} \\gets \\alpha I_1 + (1-\\alpha) I_2$ and $\\alpha \\in [0, 1]$.\nCutmix (\nYun et al., 2019\n): Cutmix does region-level mixture by generating a new example by combining a local region of one image with the rest of the other image. $I_\\text{cutmix} \\gets \\mathbf{M}_b \\odot I_1 + (1-\\mathbf{M}_b) \\odot I_2$, where $\\mathbf{M}_b \\in \\{0, 1\\}^I$ is a binary mask and $\\odot$ is element-wise multiplication. It is equivalent to filling the cutout (\nDeVries & Taylor 2017\n) region with the same region from another image.\nMoCHi (“Mixing of Contrastive Hard Negatives”;\nKalantidis et al. 2020\n): Given a query $\\mathbf{q}$, MoCHi maintains a queue of $K$ negative features $Q=\\{\\mathbf{n}_1, \\dots, \\mathbf{n}_K \\}$ and sorts these negative features by similarity to the query, $\\mathbf{q}^\\top \\mathbf{n}$, in descending order. The first $N$ items in the queue are considered as the hardest negatives, $Q^N$. Then synthetic hard examples can be generated by $\\mathbf{h} = \\tilde{\\mathbf{h}} / |\\tilde{\\mathbf{h}}|$ where $\\tilde{\\mathbf{h}} = \\alpha\\mathbf{n}_i + (1-\\alpha) \\mathbf{n}_j$ and $\\alpha \\in (0, 1)$. Even harder examples can be created by mixing with the query feature, $\\mathbf{h}’ = \\tilde{\\mathbf{h}’} / |\\tilde{\\mathbf{h}’}|_2$ where $\\tilde{\\mathbf{h}’} = \\beta\\mathbf{q} + (1-\\beta) \\mathbf{n}_j$ and $\\beta \\in (0, 0.5)$.\nParallel Augmentation\n#\nThis category of approaches produce two noise versions of one anchor image and aim to learn representation such that these two augmented samples share the same embedding.\nSimCLR\n#\nSimCLR\n(\nChen et al, 2020\n) proposed a simple framework for contrastive learning of visual representations. It learns representations for visual inputs by maximizing agreement between differently augmented views of the same sample via a contrastive loss in the latent space.\nA simple framework for contrastive learning of visual representations. (Image source:\nChen et al, 2020\n)\nRandomly sample a minibatch of $N$ samples and each sample is applied with two different data augmentation operations, resulting in $2N$ augmented samples in total.\n$$\n\\tilde{\\mathbf{x}}_i = t(\\mathbf{x}),\\quad\\tilde{\\mathbf{x}}_j = t'(\\mathbf{x}),\\quad t, t' \\sim \\mathcal{T}\n$$\nwhere two separate data augmentation operators, $t$ and $t’$, are sampled from the same family of augmentations $\\mathcal{T}$. Data augmentation includes random crop, resize with random flip, color distortions, and Gaussian blur.\nGiven one positive pair, other $2(N-1)$ data points are treated as negative samples. The representation is produced by a base encoder $f(.)$:\n$$\n\\mathbf{h}_i = f(\\tilde{\\mathbf{x}}_i),\\quad \\mathbf{h}_j = f(\\tilde{\\mathbf{x}}_j)\n$$\nThe contrastive learning loss is defined using cosine similarity $\\text{sim}(.,.)$. Note that the loss operates on an extra projection layer of the representation $g(.)$ rather than on the representation space directly. But only the representation $\\mathbf{h}$ is used for downstream tasks.\n$$\n\\begin{aligned}\n\\mathbf{z}_i &= g(\\mathbf{h}_i),\\quad\n\\mathbf{z}_j = g(\\mathbf{h}_j) \\\\\n\\mathcal{L}_\\text{SimCLR}^{(i,j)} &= - \\log\\frac{\\exp(\\text{sim}(\\mathbf{z}_i, \\mathbf{z}_j) / \\tau)}{\\sum_{k=1}^{2N} \\mathbb{1}_{[k \\neq i]} \\exp(\\text{sim}(\\mathbf{z}_i, \\mathbf{z}_k) / \\tau)}\n\\end{aligned}\n$$\nwhere $\\mathbb{1}_{[k \\neq i]}$ is an indicator function: 1 if $k\\neq i$ 0 otherwise.\nSimCLR needs a large batch size to incorporate enough negative samples to achieve good performance.\nThe algorithm for SimCLR. (Image source:\nChen et al, 2020\n).\nBarlow Twins\n#\nBarlow Twins\n(\nZbontar et al. 2021\n) feeds two distorted versions of samples into the same network to extract features and learns to make the\ncross-correlation matrix\nbetween these two groups of output features close to the identity. The goal is to keep the representation vectors of different distorted versions of one sample similar, while minimizing the redundancy between these vectors.\nIllustration of Barlow Twins learning pipeline. (Image source:\nZbontar et al. 2021\n).\nLet $\\mathcal{C}$ be a cross-correlation matrix computed between outputs from two identical networks along the batch dimension. $\\mathcal{C}$ is a square matrix with the size same as the feature network’s output dimensionality. Each entry in the matrix $\\mathcal{C}_{ij}$ is the cosine similarity between network output vector dimension at index $i, j$ and batch index $b$, $\\mathbf{z}_{b,i}^A$ and $\\mathbf{z}_{b,j}^B$, with a value between -1 (i.e. perfect anti-correlation) and 1 (i.e. perfect correlation).\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{BT} &= \\underbrace{\\sum_i (1-\\mathcal{C}_{ii})^2}_\\text{invariance term} + \\lambda \\underbrace{\\sum_i\\sum_{i\\neq j} \\mathcal{C}_{ij}^2}_\\text{redundancy reduction term} \\\\ \\text{where } \\mathcal{C}_{ij} &= \\frac{\\sum_b \\mathbf{z}^A_{b,i} \\mathbf{z}^B_{b,j}}{\\sqrt{\\sum_b (\\mathbf{z}^A_{b,i})^2}\\sqrt{\\sum_b (\\mathbf{z}^B_{b,j})^2}}\n\\end{aligned}\n$$\nBarlow Twins is competitive with SOTA methods for self-supervised learning. It naturally avoids trivial constants (i.e. collapsed representations), and is robust to different training batch sizes.\nAlgorithm of Barlow Twins in Pytorch style pseudo code. (Image source:\nZbontar et al. 2021\n).\nBYOL\n#\nDifferent from the above approaches, interestingly,\nBYOL\n(Bootstrap Your Own Latent;\nGrill, et al 2020\n) claims to achieve a new state-of-the-art results\nwithout using negative samples\n. It relies on two neural networks, referred to as\nonline\nand\ntarget\nnetworks that interact and learn from each other. The target network (parameterized by $\\xi$) has the same architecture as the online one (parameterized by $\\theta$), but with polyak averaged weights, $\\xi \\leftarrow \\tau \\xi + (1-\\tau) \\theta$.\nThe goal is to learn a presentation $y$ that can be used in downstream tasks. The online network parameterized by $\\theta$ contains:\nAn encoder $f_\\theta$;\nA projector $g_\\theta$;\nA predictor $q_\\theta$.\nThe target network has the same network architecture, but with different parameter $\\xi$, updated by polyak averaging $\\theta$: $\\xi \\leftarrow \\tau \\xi + (1-\\tau) \\theta$.\nThe model architecture of BYOL. After training, we only care about $f\\_\\theta$ for producing representation, $y=f\\_\\theta(x)$, and everything else is discarded. $\\text{sg}$ means stop gradient. (Image source:\nGrill, et al 2020\n)\nGiven an image $\\mathbf{x}$, the BYOL loss is constructed as follows:\nCreate two augmented views: $\\mathbf{v}=t(\\mathbf{x}); \\mathbf{v}’=t’(\\mathbf{x})$ with augmentations sampled $t \\sim \\mathcal{T}, t’ \\sim \\mathcal{T}’$;\nThen they are encoded into representations, $\\mathbf{y}_\\theta=f_\\theta(\\mathbf{v}), \\mathbf{y}’=f_\\xi(\\mathbf{v}’)$;\nThen they are projected into latent variables, $\\mathbf{z}_\\theta=g_\\theta(\\mathbf{y}_\\theta), \\mathbf{z}’=g_\\xi(\\mathbf{y}’)$;\nThe online network outputs a prediction $q_\\theta(\\mathbf{z}_\\theta)$;\nBoth $q_\\theta(\\mathbf{z}_\\theta)$ and $\\mathbf{z}’$ are L2-normalized, giving us $\\bar{q}_\\theta(\\mathbf{z}_\\theta) = q_\\theta(\\mathbf{z}_\\theta) / | q_\\theta(\\mathbf{z}_\\theta) |$ and $\\bar{\\mathbf{z}’} = \\mathbf{z}’ / |\\mathbf{z}’|$;\nThe loss $\\mathcal{L}^\\text{BYOL}_\\theta$ is MSE between L2-normalized prediction $\\bar{q}_\\theta(\\mathbf{z})$ and $\\bar{\\mathbf{z}’}$;\nThe other symmetric loss $\\tilde{\\mathcal{L}}^\\text{BYOL}_\\theta$ can be generated by switching $\\mathbf{v}’$ and $\\mathbf{v}$; that is, feeding $\\mathbf{v}’$ to online network and $\\mathbf{v}$ to target network.\nThe final loss is $\\mathcal{L}^\\text{BYOL}_\\theta + \\tilde{\\mathcal{L}}^\\text{BYOL}_\\theta$ and only parameters $\\theta$ are optimized.\nUnlike most popular contrastive learning based approaches, BYOL does not use negative pairs. Most bootstrapping approaches rely on pseudo-labels or cluster indices, but BYOL directly boostrapps the latent representation.\nIt is quite interesting and surprising that\nwithout\nnegative samples, BYOL still works well. Later I ran into this\npost\nby Abe Fetterman & Josh Albrecht, they highlighted two surprising findings while they were trying to reproduce BYOL:\nBYOL generally performs no better than random when\nbatch normalization is removed\n.\nThe presence of batch normalization implicitly causes a form of contrastive learning.\nThey believe that using negative samples is important for avoiding model collapse (i.e. what if you use all-zeros representation for every data point?). Batch normalization injects dependency on negative samples\ninexplicitly\nbecause no matter how similar a batch of inputs are, the values are re-distributed (spread out $\\sim \\mathcal{N}(0, 1$) and therefore batch normalization prevents model collapse. Strongly recommend you to read the\nfull article\nif you are working in this area.\nMemory Bank\n#\nComputing embeddings for a large number of negative samples in every batch is extremely expensive. One common approach is to store the representation in memory to trade off data staleness for cheaper compute.\nInstance Discrimination with Memoy Bank\n#\nInstance contrastive learning\n(\nWu et al, 2018\n) pushes the class-wise supervision to the extreme by considering each instance as\na distinct class of its own\n. It implies that the number of “classes” will be the same as the number of samples in the training dataset. Hence, it is unfeasible to train a softmax layer with these many heads, but instead it can be approximated by\nNCE\n.\nThe training pipeline of instance-level contrastive learning. The learned embedding is L2-normalized. (Image source:\nWu et al, 2018\n)\nLet $\\mathbf{v} = f_\\theta(x)$ be an embedding function to learn and the vector is normalized to have $|\\mathbf{v}|=1$. A non-parametric classifier predicts the probability of a sample $\\mathbf{v}$ belonging to class $i$ with a temperature parameter $\\tau$:\n$$\nP(C=i\\vert \\mathbf{v}) = \\frac{\\exp(\\mathbf{v}_i^\\top \\mathbf{v} / \\tau)}{\\sum_{j=1}^n \\exp(\\mathbf{v}_j^\\top \\mathbf{v} / \\tau)}\n$$\nInstead of computing the representations for all the samples every time, they implement an\nMemory Bank\nfor storing sample representation in the database from past iterations. Let $V=\\{ \\mathbf{v}_i \\}$ be the memory bank and $\\mathbf{f}_i = f_\\theta(\\mathbf{x}_i)$ be the feature generated by forwarding the network. We can use the representation from the memory bank $\\mathbf{v}_i$ instead of the feature forwarded from the network $\\mathbf{f}_i$ when comparing pairwise similarity.\nThe denominator theoretically requires access to the representations of all the samples, but that is too expensive in practice. Instead we can estimate it via Monte Carlo approximation using a random subset of $M$ indices $\\{j_k\\}_{k=1}^M$.\n$$\nP(i\\vert \\mathbf{v}) \n= \\frac{\\exp(\\mathbf{v}^\\top \\mathbf{f}_i / \\tau)}{\\sum_{j=1}^N \\exp(\\mathbf{v}_j^\\top \\mathbf{f}_i / \\tau)}\n\\simeq \\frac{\\exp(\\mathbf{v}^\\top \\mathbf{f}_i / \\tau)}{\\frac{N}{M} \\sum_{k=1}^M \\exp(\\mathbf{v}_{j_k}^\\top \\mathbf{f}_i / \\tau)}\n$$\nBecause there is only one instance per class, the training is unstable and fluctuates a lot. To improve the training smoothness, they introduced an extra term for positive samples in the loss function based on the\nproximal optimization method\n. The final NCE loss objective looks like:\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{instance} &= - \\mathbb{E}_{P_d}\\big[\\log h(i, \\mathbf{v}^{(t-1)}_i) - \\lambda \\|\\mathbf{v}^{(t)}_i - \\mathbf{v}^{(t-1)}_i\\|^2_2\\big] - M\\mathbb{E}_{P_n}\\big[\\log(1 - h(i, \\mathbf{v}'^{(t-1)})\\big] \\\\\nh(i, \\mathbf{v}) &= \\frac{P(i\\vert\\mathbf{v})}{P(i\\vert\\mathbf{v}) + MP_n(i)} \\text{ where the noise distribution is uniform }P_n = 1/N\n\\end{aligned}\n$$\nwhere $\\{ \\mathbf{v}^{(t-1)} \\}$ are embeddings stored in the memory bank from the previous iteration. The difference between iterations $|\\mathbf{v}^{(t)}_i - \\mathbf{v}^{(t-1)}_i|^2_2$ will gradually vanish as the learned embedding converges.\nMoCo & MoCo-V2\n#\nMomentum Contrast\n(\nMoCo\n;\nHe et al, 2019\n) provides a framework of unsupervised learning visual representation as a\ndynamic dictionary look-up\n. The dictionary is structured as a large FIFO queue of encoded representations of data samples.\nGiven a query sample $\\mathbf{x}_q$, we get a query representation through an encoder $\\mathbf{q} = f_q(\\mathbf{x}_q)$. A list of key representations $\\{\\mathbf{k}_1, \\mathbf{k}_2, \\dots \\}$ in the dictionary are encoded by a momentum encoder $\\mathbf{k}_i = f_k (\\mathbf{x}^k_i)$. Let’s assume among them there is a single\npositive\nkey $\\mathbf{k}^+$ in the dictionary that matches $\\mathbf{q}$. In the paper, they create $\\mathbf{k}^+$ using a noise copy of $\\mathbf{x}_q$ with different\naugmentation\n. Then the\nInfoNCE\ncontrastive loss with temperature $\\tau$ is used over one positive and $N-1$ negative samples:\n$$\n\\mathcal{L}_\\text{MoCo} = - \\log \\frac{\\exp(\\mathbf{q} \\cdot \\mathbf{k}^+ / \\tau)}{\\sum_{i=1}^N \\exp(\\mathbf{q} \\cdot \\mathbf{k}_i / \\tau)}\n$$\nCompared to the\nmemory bank\n, a queue-based dictionary in MoCo enables us to reuse representations of immediately preceding mini-batches of data.\nThe MoCo dictionary is not differentiable as a queue, so we cannot rely on back-propagation to update the key encoder $f_k$. One naive way might be to use the same encoder for both $f_q$ and $f_k$. Differently, MoCo proposed to use a momentum-based update with a momentum coefficient $m \\in [0, 1)$. Say, the parameters of $f_q$ and $f_k$ are labeled as $\\theta_q$ and $\\theta_k$, respectively.\n$$\n\\theta_k \\leftarrow m \\theta_k + (1-m) \\theta_q\n$$\nIllustration of how Momentum Contrast (MoCo) learns visual representations. (Image source:\nHe et al, 2019\n)\nThe advantage of MoCo compared to\nSimCLR\nis that MoCo decouples the batch size from the number of negatives, but SimCLR requires a large batch size in order to have enough negative samples and suffers performance drops when their batch size is reduced.\nTwo designs in SimCLR, namely, (1) an MLP projection head and (2) stronger data augmentation, are proved to be very efficient.\nMoCo V2\n(\nChen et al, 2020\n) combined these two designs, achieving even better transfer performance with no dependency on a very large batch size.\nCURL\n#\nCURL\n(\nSrinivas, et al. 2020\n) applies the above ideas in\nReinforcement Learning\n. It learns a visual representation for RL tasks by matching embeddings of two data-augmented versions, $o_q$ and $o_k$, of the raw observation $o$ via contrastive loss. CURL primarily relies on random crop data augmentation. The key encoder is implemented as a momentum encoder with weights as EMA of the query encoder weights, same as in\nMoCo\n.\nOne significant difference between RL and supervised visual tasks is that RL depends on\ntemporal consistency\nbetween consecutive frames. Therefore, CURL applies augmentation consistently on each stack of frames to retain information about the temporal structure of the observation.\nThe architecture of CURL. (Image source:\nSrinivas, et al. 2020\n)\nFeature Clustering\n#\nDeepCluster\n#\nDeepCluster\n(\nCaron et al. 2018\n) iteratively clusters features via k-means and uses cluster assignments as pseudo labels to provide supervised signals.\nIllustration of DeepCluster method which iteratively clusters deep features and uses the cluster assignments as pseudo-labels. (Image source:\nCaron et al. 2018\n)\nIn each iteration, DeepCluster clusters data points using the prior representation and then produces the new cluster assignments as the classification targets for the new representation. However this iterative process is prone to trivial solutions. While avoiding the use of negative pairs, it requires a costly clustering phase and specific precautions to avoid collapsing to trivial solutions.\nSwAV\n#\nSwAV\n(\nSwapping Assignments between multiple Views\n;\nCaron et al. 2020\n) is an online contrastive learning algorithm. It computes a code from an augmented version of the image and tries to predict this code using another augmented version of the same image.\nComparison of SwAV and [contrastive instance learning](#instance-discrimination-with-memoy-bank). (Image source:\nCaron et al. 2020\n)\nGiven features of images with two different augmentations, $\\mathbf{z}_t$ and $\\mathbf{z}_s$, SwAV computes corresponding codes $\\mathbf{q}_t$ and $\\mathbf{q}_s$ and the loss quantifies the fit by swapping two codes using $\\ell(.)$ to measure the fit between a feature and a code.\n$$\n\\mathcal{L}_\\text{SwAV}(\\mathbf{z}_t, \\mathbf{z}_s) = \\ell(\\mathbf{z}_t, \\mathbf{q}_s) + \\ell(\\mathbf{z}_s, \\mathbf{q}_t)\n$$\nThe swapped fit prediction depends on the cross entropy between the predicted code and a set of $K$ trainable prototype vectors $\\mathbf{C} = \\{\\mathbf{c}_1, \\dots, \\mathbf{c}_K\\}$. The prototype vector matrix is shared across different batches and represents\nanchor clusters\nthat each instance should be clustered to.\n$$\n\\ell(\\mathbf{z}_t, \\mathbf{q}_s) = - \\sum_k \\mathbf{q}^{(k)}_s\\log\\mathbf{p}^{(k)}_t \\text{ where } \\mathbf{p}^{(k)}_t = \\frac{\\exp(\\mathbf{z}_t^\\top\\mathbf{c}_k / \\tau)}{\\sum_{k'}\\exp(\\mathbf{z}_t^\\top \\mathbf{c}_{k'} / \\tau)}\n$$\nIn a mini-batch containing $B$ feature vectors $\\mathbf{Z} = [\\mathbf{z}_1, \\dots, \\mathbf{z}_B]$, the mapping matrix between features and prototype vectors is defined as $\\mathbf{Q} = [\\mathbf{q}_1, \\dots, \\mathbf{q}_B] \\in \\mathbb{R}_+^{K\\times B}$. We would like to maximize the similarity between the features and the prototypes:\n$$\n\\begin{aligned}\n\\max_{\\mathbf{Q}\\in\\mathcal{Q}} &\\text{Tr}(\\mathbf{Q}^\\top \\mathbf{C}^\\top \\mathbf{Z}) + \\varepsilon \\mathcal{H}(\\mathbf{Q}) \\\\\n\\text{where }\\mathcal{Q} &= \\big\\{ \\mathbf{Q} \\in \\mathbb{R}_{+}^{K \\times B} \\mid \\mathbf{Q}\\mathbf{1}_B = \\frac{1}{K}\\mathbf{1}_K, \\mathbf{Q}^\\top\\mathbf{1}_K = \\frac{1}{B}\\mathbf{1}_B \\big\\}\n\\end{aligned}\n$$\nwhere $\\mathcal{H}$ is the entropy, $\\mathcal{H}(\\mathbf{Q}) = - \\sum_{ij} \\mathbf{Q}_{ij} \\log \\mathbf{Q}_{ij}$, controlling the smoothness of the code. The coefficient $\\epsilon$ should not be too large; otherwise, all the samples will be assigned uniformly to all the clusters. The candidate set of solutions for $\\mathbf{Q}$ requires every mapping matrix to have each row sum up to $1/K$ and each column to sum up to $1/B$, enforcing that each prototype gets selected at least $B/K$ times on average.\nSwAV relies on the iterative Sinkhorn-Knopp algorithm (\nCuturi 2013\n) to find the solution for $\\mathbf{Q}$.\nWorking with Supervised Datasets\n#\nCLIP\n#\nCLIP\n(\nContrastive Language-Image Pre-training\n;\nRadford et al. 2021\n) jointly trains a text encoder and an image feature extractor over the pretraining task that predicts which caption goes with which image.\nIllustration of CLIP contrastive pre-training over text-image pairs. (Image source:\nRadford et al. 2021\n)\nGiven a batch of $N$ (image, text) pairs, CLIP computes the dense cosine similarity matrix between all $N\\times N$ possible (image, text) candidates within this batch. The text and image encoders are jointly trained to maximize the similarity between $N$ correct pairs of (image, text) associations while minimizing the similarity for $N(N-1)$ incorrect pairs via a symmetric cross entropy loss over the dense matrix.\nSee the numy-like pseudo code for CLIP in\nCLIP algorithm in Numpy style pseudo code. (Image source:\nRadford et al. 2021\n)\nCompared to other methods above for learning good visual representation, what makes CLIP really special is\n“the appreciation of using natural language as a training signal”\n. It does demand access to supervised dataset in which we know which text matches which image. It is trained on 400 million (text, image) pairs, collected from the Internet. The query list contains all the words occurring at least 100 times in the English version of Wikipedia. Interestingly, they found that Transformer-based language models are 3x slower than a bag-of-words (BoW) text encoder at zero-shot ImageNet classification. Using contrastive objective instead of trying to predict the exact words associated with images (i.e. a method commonly adopted by image caption prediction tasks) can further improve the data efficiency another 4x.\nUsing bag-of-words text encoding and contrastive training objectives can bring in multiple folds of data efficiency improvement. (Image source:\nRadford et al. 2021\n)\nCLIP produces good visual representation that can non-trivially transfer to many CV benchmark datasets, achieving results competitive with supervised baseline. Among tested transfer tasks, CLIP struggles with very fine-grained classification, as well as abstract or systematic tasks such as counting the number of objects. The transfer performance of CLIP models is smoothly correlated with the amount of model compute.\nSupervised Contrastive Learning\n#\nThere are several known issues with cross entropy loss, such as the lack of robustness to noisy labels and the possibility of poor margins. Existing improvement for cross entropy loss involves the curation of better training data, such as label smoothing and data augmentation.\nSupervised Contrastive Loss\n(\nKhosla et al. 2021\n) aims to leverage label information more effectively than cross entropy, imposing that normalized embeddings from the same class are closer together than embeddings from different classes.\nSupervised vs self-supervised contrastive losses. Supervised contrastive learning considers different samples from the same class as positive examples, in addition to augmented versions. (Image source:\nKhosla et al. 2021\n)\nGiven a set of randomly sampled $n$ (image, label) pairs, $\\{\\mathbf{x}_i, y_i\\}_{i=1}^n$, $2n$ training pairs can be created by applying two random augmentations of every sample, $\\{\\tilde{\\mathbf{x}}_i, \\tilde{y}_i\\}_{i=1}^{2n}$.\nSupervised contrastive loss $\\mathcal{L}_\\text{supcon}$ utilizes multiple positive and negative samples, very similar to\nsoft nearest-neighbor loss\n:\n$$\n\\mathcal{L}_\\text{supcon} = - \\sum_{i=1}^{2n} \\frac{1}{2 \\vert N_i \\vert - 1} \\sum_{j \\in N(y_i), j \\neq i} \\log \\frac{\\exp(\\mathbf{z}_i \\cdot \\mathbf{z}_j / \\tau)}{\\sum_{k \\in I, k \\neq i}\\exp({\\mathbf{z}_i \\cdot \\mathbf{z}_k / \\tau})}\n$$\nwhere $\\mathbf{z}_k=P(E(\\tilde{\\mathbf{x}_k}))$, in which $E(.)$ is an encoder network (augmented image mapped to vector) $P(.)$ is a projection network (one vector mapped to another). $N_i= \\{j \\in I: \\tilde{y}_j = \\tilde{y}_i \\}$ contains a set of indices of samples with label $y_i$. Including more positive samples into the set $N_i$ leads to improved results.\nAccording to their experiments, supervised contrastive loss:\ndoes outperform the base cross entropy, but only by a small amount.\noutperforms the cross entropy on robustness benchmark (ImageNet-C, which applies common naturally occuring perturbations such as noise, blur and contrast changes to the ImageNet dataset).\nis less sensitive to hyperparameter changes.\nLanguage: Sentence Embedding\n#\nIn this section, we focus on how to learn sentence embedding.\nText Augmentation\n#\nMost contrastive methods in vision applications depend on creating an augmented version of each image. However, it is more challenging to construct text augmentation which does not alter the semantics of a sentence. In this section we look into three approaches for augmenting text sequences, including lexical edits, back-translation and applying cutoff or dropout.\nLexical Edits\n#\nEDA\n(\nEasy Data Augmentation\n;\nWei & Zou 2019\n) defines a set of simple but powerful operations for text augmentation. Given a sentence, EDA randomly chooses and applies one of four simple operations:\nSynonym replacement (SR): Replace $n$ random non-stop words with their synonyms.\nRandom insertion (RI): Place a random synonym of a randomly selected non-stop word in the sentence at a random position.\nRandom swap (RS): Randomly swap two words and repeat $n$ times.\nRandom deletion (RD): Randomly delete each word in the sentence with probability $p$.\nwhere $p=\\alpha$ and $n=\\alpha \\times \\text{sentence_length}$, with the intuition that longer sentences can absorb more noise while maintaining the original label. The hyperparameter $\\alpha$ roughly indicates the percent of words in one sentence that may be changed by one augmentation.\nEDA is shown to improve the classification accuracy on several classification benchmark datasets compared to baseline without EDA. The performance lift is more significant on a smaller training set. All the four operations in EDA help improve the classification accuracy, but get to optimal at different $\\alpha$’s.\nEDA leads to performance improvement on several classification benchmarks. (Image source:\nWei & Zou 2019\n)\nIn\nContextual Augmentation\n(\nSosuke Kobayashi, 2018\n), new substitutes for word $w_i$ at position $i$ can be smoothly sampled from a given probability distribution, $p(.\\mid S\\setminus\\{w_i\\})$, which is predicted by a bidirectional LM like BERT.\nBack-translation\n#\nCERT\n(\nContrastive self-supervised Encoder Representations from Transformers\n;\nFang et al. (2020)\n;\ncode\n) generates augmented sentences via\nback-translation\n. Various translation models for different languages can be employed for creating different versions of augmentations. Once we have a noise version of text samples, many contrastive learning frameworks introduced above, such as\nMoCo\n, can be used to learn sentence embedding.\nDropout and Cutoff\n#\nShen et al. (2020)\nproposed to apply\nCutoff\nto text augmentation, inspired by\ncross-view training\n. They proposed three cutoff augmentation strategies:\nToken cutoff\nremoves the information of a few selected tokens. To make sure there is no data leakage, corresponding tokens in the input, positional and other relevant embedding matrices should all be zeroed out.,\nFeature cutoff\nremoves a few feature columns.\nSpan cutoff\nremoves a continuous chunk of texts.\nSchematic illustration of token, feature and span cutoff augmentation strategies. (Image source:\nShen et al. 2020\n)\nMultiple augmented versions of one sample can be created. When training,\nShen et al. (2020)\napplied an additional KL-divergence term to measure the consensus between predictions from different augmented samples.\nSimCSE\n(\nGao et al. 2021\n;\ncode\n) learns from unsupervised data by predicting a sentence from itself with only\ndropout\nnoise. In other words, they treat dropout as data augmentation for text sequences. A sample is simply fed into the encoder twice with different dropout masks and these two versions are the positive pair where the other in-batch samples are considered as negative pairs. It feels quite similar to the cutoff augmentation, but dropout is more flexible with less well-defined semantic meaning of what content can be masked off.\nSimCSE creates augmented samples by applying different dropout masks. The supervised version leverages NLI datasets to predict positive (entailment) or negative (contradiction) given a pair of sentences. (Image source:\nGao et al. 2021\n)\nThey ran experiments on 7 STS (Semantic Text Similarity) datasets and computed cosine similarity between sentence embeddings. They also tried out an optional MLM auxiliary objective loss to help avoid catastrophic forgetting of token-level knowledge. This aux loss was found to help improve performance on transfer tasks, but a consistent drop on the main STS tasks.\nExperiment numbers on a collection of STS benchmarks with SimCES. (Image source:\nGao et al. 2021\n)\nSupervision from NLI\n#\nThe pre-trained BERT sentence embedding without any fine-tuning has been found to have poor performance for semantic similarity tasks. Instead of using the raw embeddings directly, we need to refine the embedding with further fine-tuning.\nNatural Language Inference (NLI)\ntasks are the main data sources to provide supervised signals for learning sentence embedding; such as\nSNLI\n,\nMNLI\n, and\nQQP\n.\nSentence-BERT\n#\nSBERT (Sentence-BERT)\n(\nReimers & Gurevych, 2019\n) relies on siamese and triplet network architectures to learn sentence embeddings such that the sentence similarity can be estimated by cosine similarity between pairs of embeddings. Note that learning SBERT depends on supervised data, as it is fine-tuned on several NLI datasets.\nThey experimented with a few different prediction heads on top of BERT model:\nSoftmax classification objective: The classification head of the siamese network is built on the concatenation of two embeddings $f(\\mathbf{x}), f(\\mathbf{x}’)$ and $\\vert f(\\mathbf{x}) - f(\\mathbf{x}’) \\vert$. The predicted output is $\\hat{y}=\\text{softmax}(\\mathbf{W}_t [f(\\mathbf{x}); f(\\mathbf{x}’); \\vert f(\\mathbf{x}) - f(\\mathbf{x}’) \\vert])$. They showed that the most important component is the element-wise difference $\\vert f(\\mathbf{x}) - f(\\mathbf{x}’) \\vert$.\nRegression objective: This is the regression loss on $\\cos(f(\\mathbf{x}), f(\\mathbf{x}’))$, in which the pooling strategy has a big impact. In the experiments, they observed that\nmax\nperforms much worse than\nmean\nand\nCLS\n-token.\nTriplet objective: $\\max(0, |f(\\mathbf{x}) - f(\\mathbf{x}^+)|- |f(\\mathbf{x}) - f(\\mathbf{x}^-)| + \\epsilon)$, where $\\mathbf{x}, \\mathbf{x}^+, \\mathbf{x}^-$ are embeddings of the anchor, positive and negative sentences.\nIn the experiments, which objective function works the best depends on the datasets, so there is no universal winner.\nIllustration of Sentence-BERT training framework with softmax classification head and regression head. (Image source:\nReimers & Gurevych, 2019\n)\nThe\nSentEval\nlibrary (\nConneau and Kiela, 2018\n) is commonly used for evaluating the quality of learned sentence embedding. SBERT outperformed other baselines at that time (Aug 2019) on 5 out of 7 tasks.\nThe performance of Sentence-BERT on the SentEval benchmark. (Image source:\nReimers & Gurevych, 2019\n)\nBERT-flow\n#\nThe embedding representation space is deemed\nisotropic\nif embeddings are uniformly distributed on each dimension; otherwise, it is\nanisotropic\n.\nLi et al, (2020)\nshowed that a pre-trained BERT learns a non-smooth\nanisotropic\nsemantic space of sentence embeddings and thus leads to poor performance for text similarity tasks without fine-tuning. Empirically, they observed two issues with BERT sentence embedding:\nWord frequency biases the embedding space. High-frequency words are close to the origin, but low-frequency ones are far away from the origin.\nLow-frequency words scatter sparsely. The embeddings of low-frequency words tend to be farther to their $k$-NN neighbors, while the embeddings of high-frequency words concentrate more densely.\nBERT-flow\n(\nLi et al, 2020\n;\ncode\n) was proposed to transform the embedding to a smooth and isotropic Gaussian distribution via\nnormalizing flows\n.\nIllustration of the flow-based calibration over the original sentence embedding space in BERT-flow. (Image source:\nLi et al, 2020\n)\nLet $\\mathcal{U}$ be the observed BERT sentence embedding space and $\\mathcal{Z}$ be the desired latent space which is a standard Gaussian. Thus, $p_\\mathcal{Z}$ is a Gaussian density function and $f_\\phi: \\mathcal{Z}\\to\\mathcal{U}$ is an invertible transformation:\n$$\n\\mathbf{z}\\sim p_\\mathcal{Z}(\\mathbf{z}) \\quad \n\\mathbf{u}=f_\\phi(\\mathbf{z}) \\quad\n\\mathbf{z}=f^{-1}_\\phi(\\mathbf{u}) \n$$\nA flow-based generative model learns the invertible mapping function by maximizing the likelihood of $\\mathcal{U}$’s marginal:\n$$\n\\max_\\phi\\mathbb{E}_{\\mathbf{u}=\\text{BERT}(s), s\\sim\\mathcal{D}} \\Big[ \\log p_\\mathcal{Z}(f^{-1}_\\phi(\\mathbf{u})) + \\log\\big\\vert\\det\\frac{\\partial f^{-1}_\\phi(\\mathbf{u})}{\\partial\\mathbf{u}}\\big\\vert \\Big]\n$$\nwhere $s$ is a sentence sampled from the text corpus $\\mathcal{D}$. Only the flow parameters $\\phi$ are optimized while parameters in the pretrained BERT stay unchanged.\nBERT-flow was shown to improve the performance on most STS tasks either with or without supervision from NLI datasets. Because learning normalizing flows for calibration does not require labels, it can utilize the entire dataset including validation and test sets.\nWhitening Operation\n#\nSu et al. (2021)\napplied\nwhitening\noperation to improve the\nisotropy\nof the learned representation and also to reduce the dimensionality of sentence embedding.\nThey transform the mean value of the sentence vectors to 0 and the covariance matrix to the identity matrix. Given a set of samples $\\{\\mathbf{x}_i\\}_{i=1}^N$, let $\\tilde{\\mathbf{x}}_i$ and $\\tilde{\\Sigma}$ be the transformed samples and corresponding covariance matrix:\n$$\n\\begin{aligned}\n\\mu &= \\frac{1}{N}\\sum_{i=1}^N \\mathbf{x}_i \\quad \\Sigma = \\frac{1}{N}\\sum_{i=1}^N (\\mathbf{x}_i - \\mu)^\\top (\\mathbf{x}_i - \\mu) \\\\\n\\tilde{\\mathbf{x}}_i &= (\\mathbf{x}_i - \\mu)W \\quad \\tilde{\\Sigma} = W^\\top\\Sigma W = I \\text{ thus } \\Sigma = (W^{-1})^\\top W^{-1}\n\\end{aligned}\n$$\nIf we get\nSVD\ndecomposition of $\\Sigma = U\\Lambda U^\\top$, we will have $W^{-1}=\\sqrt{\\Lambda} U^\\top$ and $W=U\\sqrt{\\Lambda^{-1}}$. Note that within SVD, $U$ is an orthogonal matrix with column vectors as eigenvectors and $\\Lambda$ is a diagonal matrix with all positive elements as sorted eigenvalues.\nA dimensionality reduction strategy can be applied by only taking the first $k$ columns of $W$, named\nWhitening\n-$k$.\nPseudo code of the whitening-$k$ operation. (Image source:\nSu et al. 2021\n)\nWhitening operations were shown to outperform BERT-flow and achieve SOTA with 256 sentence dimensionality on many STS benchmarks, either with or without NLI supervision.\nUnsupervised Sentence Embedding Learning\n#\nContext Prediction\n#\nQuick-Thought (QT) vectors\n(\nLogeswaran & Lee, 2018\n) formulate sentence representation learning as a\nclassification\nproblem: Given a sentence and its context, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations (\n“cloze test”\n). Such a formulation removes the softmax output layer which causes training slowdown.\nIllustration of how Quick-Thought sentence embedding vectors are learned. (Image source:\nLogeswaran & Lee, 2018\n)\nLet $f(.)$ and $g(.)$ be two functions that encode a sentence $s$ into a fixed-length vector. Let $C(s)$ be the set of sentences in the context of $s$ and $S(s)$ be the set of candidate sentences including only one sentence $s_c \\in C(s)$ and many other non-context negative sentences. Quick Thoughts model learns to optimize the probability of predicting the only true context sentence $s_c \\in S(s)$. It is essentially NCE loss when considering the sentence $(s, s_c)$ as the positive pairs while other pairs $(s, s’)$ where $s’ \\in S(s), s’\\neq s_c$ as negatives.\n$$\n\\mathcal{L}_\\text{QT} \n= - \\sum_{s \\in \\mathcal{D}} \\sum_{s_c \\in C(s)} \\log p(s_c \\vert s, S(s)) \n= - \\sum_{s \\in \\mathcal{D}} \\sum_{s_c \\in C(s)}\\frac{\\exp(f(s)^\\top g(s_c))}{\\sum_{s'\\in S(s)} \\exp(f(s)^\\top g(s'))}\n$$\nMutual Information Maximization\n#\nIS-BERT (Info-Sentence BERT)\n(\nZhang et al. 2020\n;\ncode\n) adopts a self-supervised learning objective based on\nmutual information maximization\nto learn good sentence embeddings in the\nunsupervised\nmanners.\nIllustration of Info-Sentence BERT. (Image source:\nZhang et al. 2020\n)\nIS-BERT works as follows:\nUse BERT to encode an input sentence $s$ to a token embedding of length $l$, $\\mathbf{h}_{1:l}$.\nThen apply 1-D conv net with different kernel sizes (e.g. 1, 3, 5) to process the token embedding sequence to capture the n-gram local contextual dependencies: $\\mathbf{c}_i = \\text{ReLU}(\\mathbf{w} \\cdot \\mathbf{h}_{i:i+k-1} + \\mathbf{b})$. The output sequences are padded to stay the same sizes of the inputs.\nThe final local representation of the $i$-th token $\\mathcal{F}_\\theta^{(i)} (\\mathbf{x})$ is the concatenation of representations of different kernel sizes.\nThe global sentence representation $\\mathcal{E}_\\theta(\\mathbf{x})$ is computed by applying a mean-over-time pooling layer on the token representations $\\mathcal{F}_\\theta(\\mathbf{x}) = \\{\\mathcal{F}_\\theta^{(i)} (\\mathbf{x}) \\in \\mathbb{R}^d\\}_{i=1}^l$.\nSince the mutual information estimation is generally intractable for continuous and high-dimensional random variables, IS-BERT relies on the Jensen-Shannon estimator (\nNowozin et al., 2016\n,\nHjelm et al., 2019\n) to maximize the mutual information between $\\mathcal{E}_\\theta(\\mathbf{x})$ and $\\mathcal{F}_\\theta^{(i)} (\\mathbf{x})$.\n$$\nI^\\text{JSD}_\\omega(\\mathcal{F}_\\theta^{(i)} (\\mathbf{x}); \\mathcal{E}_\\theta(\\mathbf{x})) = \\mathbb{E}_{\\mathbf{x}\\sim P} [-\\text{sp}(-T_\\omega(\\mathcal{F}_\\theta^{(i)} (\\mathbf{x}); \\mathcal{E}_\\theta(\\mathbf{x})))] \\\\ - \\mathbb{E}_{\\mathbf{x}\\sim P, \\mathbf{x}' \\sim\\tilde{P}} [\\text{sp}(T_\\omega(\\mathcal{F}_\\theta^{(i)} (\\mathbf{x}'); \\mathcal{E}_\\theta(\\mathbf{x})))]\n$$\nwhere $T_\\omega: \\mathcal{F}\\times\\mathcal{E} \\to \\mathbb{R}$ is a learnable network with parameters $\\omega$, generating discriminator scores. The negative sample $\\mathbf{x}’$ is sampled from the distribution $\\tilde{P}=P$. And $\\text{sp}(x)=\\log(1+e^x)$ is the softplus activation function.\nThe unsupervised numbers on SentEval with IS-BERT outperforms most of the unsupervised baselines (Sep 2020), but unsurprisingly weaker than supervised runs. When using labelled NLI datasets, IS-BERT produces results comparable with SBERT (See Fig. 25 & 30).\nThe performance of IS-BERT on the SentEval benchmark. (Image source:\nZhang et al. 2020\n)\nCitation\n#\nCited as:\nWeng, Lilian. (May 2021). Contrastive representation learning. Lil’Log. https://lilianweng.github.io/posts/2021-05-31-contrastive/.\nOr\n@article{weng2021contrastive,\n title = \"Contrastive Representation Learning\",\n author = \"Weng, Lilian\",\n journal = \"lilianweng.github.io\",\n year = \"2021\",\n month = \"May\",\n url = \"https://lilianweng.github.io/posts/2021-05-31-contrastive/\"\n}\nReferences\n#\n[1] Sumit Chopra, Raia Hadsell and Yann LeCun.\n“Learning a similarity metric discriminatively, with application to face verification.”\nCVPR 2005.\n[2] Florian Schroff, Dmitry Kalenichenko and James Philbin.\n“FaceNet: A Unified Embedding for Face Recognition and Clustering.”\nCVPR 2015.\n[3] Hyun Oh Song et al.\n“Deep Metric Learning via Lifted Structured Feature Embedding.”\nCVPR 2016. [\ncode\n]\n[4] Ruslan Salakhutdinov and Geoff Hinton.\n“Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure”\nAISTATS 2007.\n[5] Michael Gutmann and Aapo Hyvärinen.\n“Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.”\nAISTATS 2010.\n[6] Kihyuk Sohn et al.\n“Improved Deep Metric Learning with Multi-class N-pair Loss Objective”\nNIPS 2016.\n[7] Nicholas Frosst, Nicolas Papernot and Geoffrey Hinton.\n“Analyzing and Improving Representations with the Soft Nearest Neighbor Loss.”\nICML 2019\n[8] Tongzhou Wang and Phillip Isola.\n“Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere.”\nICML 2020. [\ncode\n]\n[9] Zhirong Wu et al.\n“Unsupervised feature learning via non-parametric instance-level discrimination.”\nCVPR 2018.\n[10] Ekin D. Cubuk et al.\n“AutoAugment: Learning augmentation policies from data.”\narXiv preprint arXiv:1805.09501 (2018).\n[11] Daniel Ho et al.\n“Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules.”\nICML 2019.\n[12] Ekin D. Cubuk & Barret Zoph et al.\n“RandAugment: Practical automated data augmentation with a reduced search space.”\narXiv preprint arXiv:1909.13719 (2019).\n[13] Hongyi Zhang et al.\n“mixup: Beyond Empirical Risk Minimization.”\nICLR 2017.\n[14] Sangdoo Yun et al.\n“CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features.”\nICCV 2019.\n[15] Yannis Kalantidis et al.\n“Mixing of Contrastive Hard Negatives”\nNeuriPS 2020.\n[16] Ashish Jaiswal et al.\n“A Survey on Contrastive Self-Supervised Learning.”\narXiv preprint arXiv:2011.00362 (2021)\n[17] Jure Zbontar et al.\n“Barlow Twins: Self-Supervised Learning via Redundancy Reduction.”\narXiv preprint arXiv:2103.03230 (2021) [\ncode\n]\n[18] Alec Radford, et al.\n“Learning Transferable Visual Models From Natural Language Supervision”\narXiv preprint arXiv:2103.00020 (2021)\n[19] Mathilde Caron et al.\n“Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (SwAV).”\nNeuriPS 2020.\n[20] Mathilde Caron et al.\n“Deep Clustering for Unsupervised Learning of Visual Features.”\nECCV 2018.\n[21] Prannay Khosla et al.\n“Supervised Contrastive Learning.”\nNeurIPS 2020.\n[22] Aaron van den Oord, Yazhe Li & Oriol Vinyals.\n“Representation Learning with Contrastive Predictive Coding”\narXiv preprint arXiv:1807.03748 (2018).\n[23] Jason Wei and Kai Zou.\n“EDA: Easy data augmentation techniques for boosting performance on text classification tasks.”\nEMNLP-IJCNLP 2019.\n[24] Sosuke Kobayashi.\n“Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations.”\nNAACL 2018\n[25] Hongchao Fang et al.\n“CERT: Contrastive self-supervised learning for language understanding.”\narXiv preprint arXiv:2005.12766 (2020).\n[26] Dinghan Shen et al.\n“A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation.”\narXiv preprint arXiv:2009.13818 (2020) [\ncode\n]\n[27] Tianyu Gao et al.\n“SimCSE: Simple Contrastive Learning of Sentence Embeddings.”\narXiv preprint arXiv:2104.08821 (2020). [\ncode\n]\n[28] Nils Reimers and Iryna Gurevych.\n“Sentence-BERT: Sentence embeddings using Siamese BERT-networks.”\nEMNLP 2019.\n[29] Jianlin Su et al.\n“Whitening sentence representations for better semantics and faster retrieval.”\narXiv preprint arXiv:2103.15316 (2021). [\ncode\n]\n[30] Yan Zhang et al.\n“An unsupervised sentence embedding method by mutual information maximization.”\nEMNLP 2020. [\ncode\n]\n[31] Bohan Li et al.\n“On the sentence embeddings from pre-trained language models.”\nEMNLP 2020.\n[32] Lajanugen Logeswaran and Honglak Lee.\n“An efficient framework for learning sentence representations.”\nICLR 2018.\n[33] Joshua Robinson, et al.\n“Contrastive Learning with Hard Negative Samples.”\nICLR 2021.\n[34] Ching-Yao Chuang et al.\n“Debiased Contrastive Learning.”\nNeuriPS 2020.", "metadata": { "title": "Contrastive Representation Learning | Lil'Log", "description": "The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.", "keywords": "representation-learning, long-read, language-model, unsupervised-learning, data-augmentation", "author": "Lilian Weng", "og:title": "Contrastive Representation Learning", "og:description": "The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.", "og:type": "article", "og:url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/", "twitter:card": "summary", "twitter:title": "Contrastive Representation Learning", "twitter:description": "The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.", "article:section": "posts", "article:published_time": "2021-05-31T00:00:00+00:00", "article:modified_time": "2021-05-31T00:00:00+00:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/c8f3f2a59bb8/code/body-000.txt", "chars": 259, "language": "text" } ], "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/triplet-loss.png", "bytes": 82669, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-c8feb8e3831d6f407222.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/triplet-loss.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/lifted-structured-loss.png", "bytes": 238558, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-864616500d13b290a57d.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/lifted-structured-loss.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-sampling-bias.png", "bytes": 618000, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-4d1eafbd2698e4b13414.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-sampling-bias.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-debias-t-SNE.png", "bytes": 447800, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-2202dbf059680ff25f64.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-debias-t-SNE.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-hard-negatives-code.png", "bytes": 331316, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-56fdc531021fd8fc266d.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/contrastive-hard-negatives-code.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCLR.png", "bytes": 92328, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-a3de29261702db65276c.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCLR.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCLR-algo.png", "bytes": 292083, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-9fbb7be242e78cf4064e.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCLR-algo.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/barlow-twins.png", "bytes": 165448, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-7c6e570b2b4acaea53ee.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/barlow-twins.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/barlow-twins-algo.png", "bytes": 410905, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-0dc7fb2e267a076918a0.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/barlow-twins-algo.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/BYOL.png", "bytes": 160971, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-64c06b583b29bc8d849d.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/BYOL.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/instance-level-discrimination.png", "bytes": 575598, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-24abca83dac665bdfd2c.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/instance-level-discrimination.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/MoCo.png", "bytes": 191327, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-68a57a589ced1a8dcbeb.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/MoCo.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CURL.png", "bytes": 458778, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-dec760a7d6d875b1e5f6.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CURL.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/deepcluster.png", "bytes": 413067, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-3b05f2c9b09900dd678c.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/deepcluster.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SwAV.png", "bytes": 307781, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-9fe78703c0d6dd59901b.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SwAV.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP.png", "bytes": 395407, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-bcffab9e56e403b12eb6.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP-algo.png", "bytes": 378369, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-14fc3bb2a028bd633430.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP-algo.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP-efficiency.png", "bytes": 269518, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-1cdb06387d4aabe19e7f.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/CLIP-efficiency.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/sup-con.png", "bytes": 1172082, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-dc3294612801cee60ee0.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/sup-con.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/EDA-exp1.png", "bytes": 299081, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-4ecf4ae1a9c6c75e0b98.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/EDA-exp1.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/text-cutoff.png", "bytes": 102477, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-2f938fd27f3112a7668b.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/text-cutoff.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCSE.png", "bytes": 302804, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-476b14449fa38214486c.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCSE.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCSE-STS-exp.png", "bytes": 459618, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-d417f179aa932198443f.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SimCSE-STS-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SBERT.png", "bytes": 248555, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-61290a159906f77df170.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SBERT.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SBERT-SentEval.png", "bytes": 263986, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-25636cdf26fe48bc32a6.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/SBERT-SentEval.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/BERT-flow.png", "bytes": 127994, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-1f6149bb39a134da1e75.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/BERT-flow.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/whitening-SBERT.png", "bytes": 167429, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-75a0434d42ea9e13e5f2.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/whitening-SBERT.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/quick-thought.png", "bytes": 235880, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-0873d959165b59705852.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/quick-thought.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/IS-BERT.png", "bytes": 403903, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-7fc9d0b8fcf86efd800b.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/IS-BERT.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/IS-BERT-SentEval.png", "bytes": 237372, "reused": false, "local_path": "attachments/c8f3f2a59bb8/images/body-4a85aecfe0dc34b27289.png", "src": "https://lilianweng.github.io/posts/2021-05-31-contrastive/IS-BERT-SentEval.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://lilianweng.github.io/posts/2019-11-10-self-supervised/", "label": "self-supervised learning", "type": "a" }, { "href": "http://yann.lecun.com/exdb/publis/pdf/chopra-05.pdf", "label": "Chopra et al. 2005", "type": "a" }, { "href": "https://arxiv.org/abs/1503.03832", "label": "Schroff et al. 2015", "type": "a" }, { "href": "https://arxiv.org/abs/1511.06452", "label": "Song et al. 2015", "type": "a" }, { "href": "https://papers.nips.cc/paper/2016/hash/6b180037abbebea991d8b1232f8a8ca9-Abstract.html", "label": "Sohn 2016", "type": "a" }, { "href": "http://proceedings.mlr.press/v9/gutmann10a.html", "label": "Gutmann & Hyvarinen", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2017-10-15-word-embedding/", "label": "here", "type": "a" }, { "href": "https://arxiv.org/abs/1807.03748", "label": "van den Oord, et al. 2018", "type": "a" }, { "href": "http://proceedings.mlr.press/v2/salakhutdinov07a.html", "label": "Salakhutdinov & Hinton 2007", "type": "a" }, { "href": "https://arxiv.org/abs/1902.01889", "label": "Frosst et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/2005.10242", "label": "Wang & Isola 2020", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2020-10-29-odqa/", "label": "DPR", "type": "a" }, { "href": "https://arxiv.org/abs/2004.04906", "label": "Karpukhin et al., 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2007.00224", "label": "Chuang et al. (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/2010.04592", "label": "Robinson et al. (2021)", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2019-05-05-domain-randomization/", "label": "AutoAugment", "type": "a" }, { "href": "https://arxiv.org/abs/1805.09501", "label": "Cubuk, et al. 2018", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2020-08-06-nas/", "label": "NAS", "type": "a" }, { "href": "https://arxiv.org/abs/1909.13719", "label": "Cubuk et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1905.05393", "label": "Ho et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1711.09846", "label": "Jaderberg et al, 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1904.12848", "label": "Xie et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1710.09412", "label": "Zhang et al., 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1905.04899", "label": "Yun et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1708.04552", "label": "DeVries & Taylor 2017", "type": "a" }, { "href": "https://arxiv.org/abs/2010.01028", "label": "Kalantidis et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2002.05709", "label": "Chen et al, 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2103.03230", "label": "Zbontar et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2006.07733", "label": "Grill, et al 2020", "type": "a" }, { "href": "https://untitled-ai.github.io/understanding-self-supervised-contrastive-learning.html", "label": "post", "type": "a" }, { "href": "https://arxiv.org/abs/1805.01978v1", "label": "Wu et al, 2018", "type": "a" }, { "href": "https://web.stanford.edu/~boyd/papers/prox_algs.html", "label": "proximal optimization method", "type": "a" }, { "href": "https://arxiv.org/abs/1911.05722", "label": "He et al, 2019", "type": "a" }, { "href": "https://arxiv.org/abs/2003.04297", "label": "Chen et al, 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2004.04136", "label": "Srinivas, et al. 2020", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/", "label": "Reinforcement Learning", "type": "a" }, { "href": "https://arxiv.org/abs/1807.05520", "label": "Caron et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/2006.09882", "label": "Caron et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1306.0895", "label": "Cuturi 2013", "type": "a" }, { "href": "https://arxiv.org/abs/2103.00020", "label": "Radford et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2004.11362", "label": "Khosla et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/1901.11196", "label": "Wei & Zou 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1805.06201", "label": "Sosuke Kobayashi, 2018", "type": "a" }, { "href": "https://arxiv.org/abs/2005.12766", "label": "Fang et al. (2020)", "type": "a" }, { "href": "https://github.com/UCSD-AI4H/CERT", "label": "code", "type": "a" }, { "href": "https://arxiv.org/abs/2009.13818", "label": "Shen et al. (2020)", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2019-01-31-lm/", "label": "cross-view training", "type": "a" }, { "href": "https://arxiv.org/abs/2104.08821", "label": "Gao et al. 2021", "type": "a" }, { "href": "https://github.com/princeton-nlp/SimCSE", "label": "code", "type": "a" }, { "href": "https://nlp.stanford.edu/projects/snli/", "label": "SNLI", "type": "a" }, { "href": "https://cims.nyu.edu/~sbowman/multinli/", "label": "MNLI", "type": "a" }, { "href": "https://www.kaggle.com/c/quora-question-pairs", "label": "QQP", "type": "a" }, { "href": "https://arxiv.org/abs/1908.10084", "label": "Reimers & Gurevych, 2019", "type": "a" }, { "href": "https://github.com/facebookresearch/SentEval", "label": "SentEval", "type": "a" }, { "href": "https://arxiv.org/abs/1803.05449", "label": "Conneau and Kiela, 2018", "type": "a" }, { "href": "https://arxiv.org/abs/2011.05864", "label": "Li et al, (2020)", "type": "a" }, { "href": "https://github.com/bohanli/BERT-flow", "label": "code", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2018-10-13-flow-models/", "label": "normalizing flows", "type": "a" }, { "href": "https://arxiv.org/abs/2103.15316", "label": "Su et al. (2021)", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Singular_value_decomposition", "label": "SVD", "type": "a" }, { "href": "https://arxiv.org/abs/1803.02893", "label": "Logeswaran & Lee, 2018", "type": "a" }, { "href": "https://arxiv.org/abs/2009.12061", "label": "Zhang et al. 2020", "type": "a" }, { "href": "https://github.com/yanzhangnlp/IS-BERT", "label": "code", "type": "a" }, { "href": "https://arxiv.org/abs/1606.00709", "label": "Nowozin et al., 2016", "type": "a" }, { "href": "https://arxiv.org/abs/1808.06670", "label": "Hjelm et al., 2019", "type": "a" }, { "href": "https://github.com/rksltnl/Deep-Metric-Learning-CVPR16", "label": "code", "type": "a" }, { "href": "http://proceedings.mlr.press/v97/frosst19a.html", "label": "“Analyzing and Improving Representations with the Soft Nearest Neighbor Loss.”", "type": "a" }, { "href": "https://ssnl.github.io/hypersphere/", "label": "code", "type": "a" }, { "href": "https://arxiv.org/abs/1805.01978", "label": "“Unsupervised feature learning via non-parametric instance-level discrimination.”", "type": "a" }, { "href": "https://arxiv.org/abs/2011.00362", "label": "“A Survey on Contrastive Self-Supervised Learning.”", "type": "a" }, { "href": "https://github.com/facebookresearch/barlowtwins", "label": "code", "type": "a" }, { "href": "https://github.com/dinghanshen/cutoff", "label": "code", "type": "a" }, { "href": "https://github.com/bojone/BERT-whitening", "label": "code", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:51:20Z", "source_url": "https://lilianweng.github.io/posts/2021-05-31-contrastive/", "papers": [ { "status": "ok", "url": "http://yann.lecun.com/exdb/publis/pdf/chopra-05.pdf", "bytes": 313804, "reused": false, "local_path": "papers/repair/84881b743b9447eb9afe.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1503.03832", "bytes": 4705274, "reused": false, "local_path": "papers/repair/a71730a10b0aef666206.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1511.06452", "bytes": 9011569, "reused": false, "local_path": "papers/repair/738c3f04b7e4d7dad3a9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1807.03748", "bytes": 3182089, "reused": false, "local_path": "papers/repair/7ace1819f0b552bee2ab.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1902.01889", "bytes": 3823806, "reused": false, "local_path": "papers/repair/66ab4748bfca0238682e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2005.10242", "bytes": 4185717, "reused": false, "local_path": "papers/repair/8a0c585591f7f1918d90.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2004.04906", "bytes": 383508, "reused": true, "local_path": "papers/repair/8b76986dbe24871ab2c9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2007.00224", "bytes": 5343399, "reused": false, "local_path": "papers/repair/7dcaf70ed91e5defcf23.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2010.04592", "bytes": 2199907, "reused": false, "local_path": "papers/repair/3f84b66b72f5c381c05a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1805.09501", "bytes": 1138440, "reused": true, "local_path": "papers/repair/9a0542ec64439d672cc9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.13719", "bytes": 1263406, "reused": true, "local_path": "papers/repair/c267ff367c4770ddb6c0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.05393", "bytes": 873126, "reused": true, "local_path": "papers/repair/4634b6280b855d570ac8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1711.09846", "bytes": 2370925, "reused": true, "local_path": "papers/repair/3bbdbf9c8931b9ea9ea0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1904.12848", "bytes": 1625724, "reused": true, "local_path": "papers/repair/17af125202b45606d9ad.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1710.09412", "bytes": 740562, "reused": true, "local_path": "papers/repair/7af2d52fbf38a69ec251.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.04899", "bytes": 1682836, "reused": true, "local_path": "papers/repair/fec62c4fe8efdeebd877.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1708.04552", "bytes": 741919, "reused": true, "local_path": "papers/repair/c3d681ed19bf88f43a98.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2010.01028", "bytes": 970998, "reused": true, "local_path": "papers/repair/d4d3c2412459021d347a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.05709", "bytes": 6564895, "reused": false, "local_path": "papers/repair/702dc85ba981458cb06f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2103.03230", "bytes": 2434163, "reused": false, "local_path": "papers/repair/7f141d9ed4155ccf53f5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2006.07733", "bytes": 4837213, "reused": false, "local_path": "papers/repair/7c03b7b76959caf6d3ab.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1805.01978v1", "bytes": 4452342, "reused": false, "local_path": "papers/repair/e90b74cda724c45827a6.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1911.05722", "bytes": 593470, "reused": false, "local_path": "papers/repair/5c3c001fd908f6dae2c6.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2003.04297", "bytes": 262792, "reused": false, "local_path": "papers/repair/43fc2eaa495db084a96e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2004.04136", "bytes": 4108339, "reused": false, "local_path": "papers/repair/c9a2ff8f5b0e5380a701.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1807.05520", "bytes": 8629266, "reused": false, "local_path": "papers/repair/0a5b97b834b6a105dd9e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2006.09882", "bytes": 702914, "reused": false, "local_path": "papers/repair/0838c337bef6ccd33d62.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1306.0895", "bytes": 1004298, "reused": false, "local_path": "papers/repair/598da2f78a40c9c822ff.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2103.00020", "bytes": 6813639, "reused": true, "local_path": "papers/repair/7db0b9244bb1fd42ae98.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2004.11362", "bytes": 2197586, "reused": false, "local_path": "papers/repair/a64eec551d692894bad9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1901.11196", "bytes": 365049, "reused": true, "local_path": "papers/repair/389e34d78a0045e6d861.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1805.06201", "bytes": 722323, "reused": true, "local_path": "papers/repair/aebd3f77254c72e012a3.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2005.12766", "bytes": 412104, "reused": false, "local_path": "papers/repair/bae8e6b74f13390eb0a6.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2009.13818", "bytes": 398629, "reused": true, "local_path": "papers/repair/40a2d9b786e78e3cf30e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2104.08821", "bytes": 1672580, "reused": true, "local_path": "papers/repair/e7a40652bdd04cd4b25f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1908.10084", "bytes": 548961, "reused": true, "local_path": "papers/repair/20151599fbbbabacaa9e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1803.05449", "bytes": 1257905, "reused": false, "local_path": "papers/repair/748bb3642ea0aef5571f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2011.05864", "bytes": 408978, "reused": false, "local_path": "papers/repair/3bfe284fc1ebd8a9d3a3.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2103.15316", "bytes": 659541, "reused": false, "local_path": "papers/repair/6e8262dafcfd96e9b679.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1803.02893", "bytes": 232651, "reused": false, "local_path": "papers/repair/12ee892560b48ed47fb8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2009.12061", "bytes": 408861, "reused": false, "local_path": "papers/repair/5c6f2a504c1b0ff66969.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1606.00709", "bytes": 3615251, "reused": false, "local_path": "papers/repair/b555d3a1dac849208314.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1808.06670", "bytes": 4984410, "reused": false, "local_path": "papers/repair/ca0c4e18cba031b31bd0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1805.01978", "bytes": 4452342, "reused": false, "local_path": "papers/repair/2408633e99d87285a643.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2011.00362", "bytes": 5998084, "reused": false, "local_path": "papers/repair/20fd1312b2e030a81069.pdf" } ], "actual_video_files": [], "body_chars": 62279, "expected_body_images": 30, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/c8f3f2a59bb8/pages/source.html", "local_html": "attachments/c8f3f2a59bb8/pages/article.html", "body_base_path": "attachments/c8f3f2a59bb8/pages/", "body_markdown_path": "attachments/c8f3f2a59bb8/pages/article.md" } }, { "url": "https://lilianweng.github.io/posts/2019-01-31-lm/", "status": "partial", "article_id": "607baf97b999", "local_path": "series-follow/1602/repair/98f025266fe083b5aa56.html", "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/elmo-and-bert.png", "bytes": 929637, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/5e717a51b0a5329fa972.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/nmt-recap.png", "bytes": 268193, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/bcfb5637b30440671005.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/CoVe.png", "bytes": 183059, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/7365015a0a10cc69acd8.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ELMo-biLSTM.png", "bytes": 473871, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/9d994d94aa3f953655fe.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/CVT.png", "bytes": 244780, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/4b7c5a8a25f1aef4e33a.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/CVT-example.png", "bytes": 269806, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/7ad146452781a398f651.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ULMFiT.png", "bytes": 1588495, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/2208e66f87d45609e14a.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/OpenAI-GPT-transformer-decoder.png", "bytes": 516917, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/e495e282eb1d3f3e9144.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT-classification.png", "bytes": 51037, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/8a0f9752f056eb917bbf.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT-downstream-tasks.png", "bytes": 272909, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/b96d2353529c40b14aef.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/transformer-encoder-2.png", "bytes": 100122, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/e2ee524fbebc06b6e190.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/language-model-comparison.png", "bytes": 227696, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/5d14984c36bf66c73897.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BERT-input-embedding.png", "bytes": 87898, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/08c8fd1e0c80108aa470.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BERT-downstream-tasks.png", "bytes": 415583, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/c58896fcdd52fc06c2a0.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/T5.png", "bytes": 253294, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/7970b3c999effb6f02b6.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT3-train-data.png", "bytes": 88113, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/85fc2d78dbd4ba853f15.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT3-eval.png", "bytes": 282592, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/dd77dfe447995839038a.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/XLNet-two-stream-attention.png", "bytes": 384564, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/3be7c9763f11993ba7a6.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/XLNet-glue.png", "bytes": 76634, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/e529afcc9a2bd0fb2fae.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BART.png", "bytes": 180008, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/8045a59ac9bcdcc697fc.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BART-perf.png", "bytes": 258219, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/c0273e122159d4c19386.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ELECTRA-overview.png", "bytes": 88004, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/8bf9b1628b286eb863de.png" }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ELECTRA-perf.png", "bytes": 138525, "reused": false, "content_type": "image/png", "local_path": "series-follow/1602/repair/assets/76b71f8459361bdc98c5.png" } ], "article": { "id": "607baf97b999", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/", "source_excel_row": 1602, "source_url": "https://lilianweng.github.io/posts/2019-01-31-lm/", "is_paid_flag": "", "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/elmo-and-bert.png", "bytes": 929637, "reused": true, "local_path": "attachments/607baf97b999/images/body-5e717a51b0a5329fa972.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/elmo-and-bert.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/nmt-recap.png", "bytes": 268193, "reused": true, "local_path": "attachments/607baf97b999/images/body-bcfb5637b30440671005.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/nmt-recap.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/CoVe.png", "bytes": 183059, "reused": true, "local_path": "attachments/607baf97b999/images/body-7365015a0a10cc69acd8.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/CoVe.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ELMo-biLSTM.png", "bytes": 473871, "reused": true, "local_path": "attachments/607baf97b999/images/body-9d994d94aa3f953655fe.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/ELMo-biLSTM.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/CVT.png", "bytes": 244780, "reused": true, "local_path": "attachments/607baf97b999/images/body-4b7c5a8a25f1aef4e33a.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/CVT.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/CVT-example.png", "bytes": 269806, "reused": true, "local_path": "attachments/607baf97b999/images/body-7ad146452781a398f651.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/CVT-example.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ULMFiT.png", "bytes": 1588495, "reused": true, "local_path": "attachments/607baf97b999/images/body-2208e66f87d45609e14a.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/ULMFiT.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/OpenAI-GPT-transformer-decoder.png", "bytes": 516917, "reused": true, "local_path": "attachments/607baf97b999/images/body-e495e282eb1d3f3e9144.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/OpenAI-GPT-transformer-decoder.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT-classification.png", "bytes": 51037, "reused": true, "local_path": "attachments/607baf97b999/images/body-8a0f9752f056eb917bbf.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT-classification.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT-downstream-tasks.png", "bytes": 272909, "reused": true, "local_path": "attachments/607baf97b999/images/body-b96d2353529c40b14aef.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT-downstream-tasks.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/transformer-encoder-2.png", "bytes": 100122, "reused": true, "local_path": "attachments/607baf97b999/images/body-e2ee524fbebc06b6e190.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/transformer-encoder-2.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/language-model-comparison.png", "bytes": 227696, "reused": true, "local_path": "attachments/607baf97b999/images/body-5d14984c36bf66c73897.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/language-model-comparison.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BERT-input-embedding.png", "bytes": 87898, "reused": true, "local_path": "attachments/607baf97b999/images/body-08c8fd1e0c80108aa470.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/BERT-input-embedding.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BERT-downstream-tasks.png", "bytes": 415583, "reused": true, "local_path": "attachments/607baf97b999/images/body-c58896fcdd52fc06c2a0.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/BERT-downstream-tasks.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/T5.png", "bytes": 253294, "reused": true, "local_path": "attachments/607baf97b999/images/body-7970b3c999effb6f02b6.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/T5.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT3-train-data.png", "bytes": 88113, "reused": true, "local_path": "attachments/607baf97b999/images/body-85fc2d78dbd4ba853f15.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT3-train-data.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT3-eval.png", "bytes": 282592, "reused": true, "local_path": "attachments/607baf97b999/images/body-dd77dfe447995839038a.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/GPT3-eval.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/XLNet-two-stream-attention.png", "bytes": 384564, "reused": true, "local_path": "attachments/607baf97b999/images/body-3be7c9763f11993ba7a6.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/XLNet-two-stream-attention.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/XLNet-glue.png", "bytes": 76634, "reused": true, "local_path": "attachments/607baf97b999/images/body-e529afcc9a2bd0fb2fae.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/XLNet-glue.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BART.png", "bytes": 180008, "reused": true, "local_path": "attachments/607baf97b999/images/body-8045a59ac9bcdcc697fc.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/BART.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/BART-perf.png", "bytes": 258219, "reused": true, "local_path": "attachments/607baf97b999/images/body-c0273e122159d4c19386.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/BART-perf.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ELECTRA-overview.png", "bytes": 88004, "reused": true, "local_path": "attachments/607baf97b999/images/body-8bf9b1628b286eb863de.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/ELECTRA-overview.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2019-01-31-lm/ELECTRA-perf.png", "bytes": 138525, "reused": true, "local_path": "attachments/607baf97b999/images/body-76b71f8459361bdc98c5.png", "src": "https://lilianweng.github.io/posts/2019-01-31-lm/ELECTRA-perf.png", "alt": "", "downloaded": true } ], "videos": [ { "url": "https://www.youtube.com/watch?v=l5einDQ-Ttc", "src": "https://www.youtube.com/watch?v=l5einDQ-Ttc", "status": "ok", "kind": "single_video", "local_path": "attachments/607baf97b999/videos/repair-l5einDQ-Ttc.mp4", "bytes": 19398754, "valid": true, "height": 1080, "width": 1920, "duration": "139.221000", "reused": false } ], "body_markdown": "[Updated on 2019-02-14: add\nULMFiT\nand\nGPT-2\n.]\n[Updated on 2020-02-29: add\nALBERT\n.]\n[Updated on 2020-10-25: add\nRoBERTa\n.]\n[Updated on 2020-12-13: add\nT5\n.]\n[Updated on 2020-12-30: add\nGPT-3\n.]\n[Updated on 2021-11-13: add\nXLNet\n,\nBART\nand\nELECTRA\n; Also updated the\nSummary\nsection.]\nI guess they are Elmo & Bert? (Image source:\nhere\n)\nWe have seen amazing progress in NLP in 2018. Large-scale pre-trained language modes like\nOpenAI GPT\nand\nBERT\nhave achieved great performance on a variety of language tasks using generic model architectures. The idea is similar to how ImageNet classification pre-training helps many vision tasks (*). Even better than vision classification pre-training, this simple and powerful approach in NLP does not require labeled data for pre-training, allowing us to experiment with increased training scale, up to our very limit.\n(*) He et al. (2018)\nfound\nthat pre-training might not be necessary for image segmentation task.\nIn my previous NLP\npost on word embedding\n, the introduced embeddings are not context-specific — they are learned based on word concurrency but not sequential context. So in two sentences, “\nI am eating an apple\n” and “\nI have an Apple phone\n”, two “apple” words refer to very different things but they would still share the same word embedding vector.\nDespite this, early adoption of word embeddings in problem-solving is to use them as additional features for an existing task-specific model and in a way the improvement is bounded.\nIn this post, we will discuss how various approaches were proposed to make embeddings dependent on context, and to make them easier and cheaper to be applied to downstream tasks in general form.\nCoVe\n#\nCoVe\n(\nMcCann et al. 2017\n), short for\nContextual Word Vectors\n, is a type of word embeddings learned by an encoder in an\nattentional seq-to-seq\nmachine translation model.\nDifferent from traditional word embeddings introduced\nhere\n, CoVe word representations are functions of the entire input sentence.\nNMT Recap\n#\nHere the Neural Machine Translation (\nNMT\n) model is composed of a standard, two-layer, bidirectional LSTM encoder and an attentional two-layer unidirectional LSTM decoder. It is pre-trained on the English-German translation task. The encoder learns and optimizes the embedding vectors of English words in order to translate them to German. With the intuition that the encoder should capture high-level semantic and syntactic meanings before transforming words into another language, the encoder output is used to provide contextualized word embeddings for various downstream language tasks.\nThe NMT base model used in CoVe.\nA sequence of $n$ words in source language (English): $x = [x_1, \\dots, x_n]$.\nA sequence of $m$ words in target language (German): $y = [y_1, \\dots, y_m]$.\nThe\nGloVe\nvectors of source words: $\\text{GloVe}(x)$.\nRandomly initialized embedding vectors of target words: $z = [z_1, \\dots, z_m]$.\nThe biLSTM encoder outputs a sequence of hidden states: $h = [h_1, \\dots, h_n] = \\text{biLSTM}(\\text{GloVe}(x))$ and $h_t = [\\overrightarrow{h}_t; \\overleftarrow{h}_t]$ where the forward LSTM computes $\\overrightarrow{h}_t = \\text{LSTM}(x_t, \\overrightarrow{h}_{t-1})$ and the backward computation gives us $\\overleftarrow{h}_t = \\text{LSTM}(x_t, \\overleftarrow{h}_{t-1})$.\nThe attentional decoder outputs a distribution over words: $p(y_t \\mid H, y_1, \\dots, y_{t-1})$ where $H$ is a stack of hidden states $\\{h\\}$ along the time dimension:\n$$\n\\begin{aligned}\n\\text{decoder hidden state: } s_t &= \\text{LSTM}([z_{t-1}; \\tilde{h}_{t-1}], s_{t-1}) \\\\\n\\text{attention weights: } \\alpha_t &= \\text{softmax}(H(W_1 s_t + b_1)) \\\\\n\\text{context-adjusted hidden state: } \\tilde{h}_t &= \\tanh(W_2[H^\\top\\alpha_t;s_t] + b_2) \\\\\n\\text{decoder output: } p(y_t\\mid H, y_1, \\dots, y_{t-1}) &= \\text{softmax}(W_\\text{out} \\tilde{h}_t + b_\\text{out})\n\\end{aligned}\n$$\nUse CoVe in Downstream Tasks\n#\nThe hidden states of NMT encoder are defined as\ncontext vectors\nfor other language tasks:\n$$\n\\text{CoVe}(x) = \\text{biLSTM}(\\text{GloVe}(x))\n$$\nThe paper proposed to use the concatenation of GloVe and CoVe for question-answering and classification tasks. GloVe learns from the ratios of global word co-occurrences, so it has no sentence context, while CoVe is generated by processing text sequences is able to capture the contextual information.\n$$\nv = [\\text{GloVe}(x); \\text{CoVe}(x)]\n$$\nGiven a downstream task, we first generate the concatenation of GloVe + CoVe vectors of input words and then feed them into the task-specific models as additional features.\nThe CoVe embeddings are generated by an encoder trained for machine translation task. The encoder can be plugged into any downstream task-specific model. (Image source:\noriginal paper\n)\nSummary\n: The limitation of CoVe is obvious: (1) pre-training is bounded by available datasets on the supervised translation task; (2) the contribution of CoVe to the final performance is constrained by the task-specific model architecture.\nIn the following sections, we will see that ELMo overcomes issue (1) by unsupervised pre-training and OpenAI GPT & BERT further overcome both problems by unsupervised pre-training + using generative model architecture for different downstream tasks.\nELMo\n#\nELMo\n, short for\nEmbeddings from Language Model\n(\nPeters, et al, 2018\n) learns contextualized word representation by pre-training a language model in an\nunsupervised\nway.\nBidirectional Language Model\n#\nThe bidirectional Language Model (\nbiLM\n) is the foundation for ELMo. While the input is a sequence of $n$ tokens, $(x_1, \\dots, x_n)$, the language model learns to predict the probability of next token given the history.\nIn the forward pass, the history contains words before the target token,\n$$\np(x_1, \\dots, x_n) = \\prod_{i=1}^n p(x_i \\mid x_1, \\dots, x_{i-1})\n$$\nIn the backward pass, the history contains words after the target token,\n$$\np(x_1, \\dots, x_n) = \\prod_{i=1}^n p(x_i \\mid x_{i+1}, \\dots, x_n)\n$$\nThe predictions in both directions are modeled by multi-layer LSTMs with hidden states $\\overrightarrow{\\mathbf{h}}_{i,\\ell}$ and $\\overleftarrow{\\mathbf{h}}_{i,\\ell}$ for input token $x_i$ at the layer level $\\ell=1,\\dots,L$.\nThe final layer’s hidden state $\\mathbf{h}_{i,L} = [\\overrightarrow{\\mathbf{h}}_{i,L}; \\overleftarrow{\\mathbf{h}}_{i,L}]$ is used to output the probabilities over tokens after softmax normalization. They share the embedding layer and the softmax layer, parameterized by $\\Theta_e$ and $\\Theta_s$ respectively.\nThe biLSTM base model of ELMo. (Image source: recreated based on the figure in [\"Neural Networks, Types, and Functional Programming\"](http://colah.github.io/posts/2015-09-NN-Types-FP/) by Christopher Olah.)\nThe model is trained to minimize the negative log likelihood (= maximize the log likelihood for true words) in both directions:\n$$\n\\begin{aligned}\n\\mathcal{L} = - \\sum_{i=1}^n \\Big( \n\\log p(x_i \\mid x_1, \\dots, x_{i-1}; \\Theta_e, \\overrightarrow{\\Theta}_\\text{LSTM}, \\Theta_s) + \\\\\n\\log p(x_i \\mid x_{i+1}, \\dots, x_n; \\Theta_e, \\overleftarrow{\\Theta}_\\text{LSTM}, \\Theta_s) \\Big)\n\\end{aligned}\n$$\nELMo Representations\n#\nOn top of a $L$-layer biLM, ELMo stacks all the hidden states across layers together by learning a task-specific linear combination. The hidden state representation for the token $x_i$ contains $2L+1$ vectors:\n$$\nR_i = \\{ \\mathbf{h}_{i,\\ell} \\mid \\ell = 0, \\dots, L \\}\n$$\nwhere $\\mathbf{h}_{0, \\ell}$ is the embedding layer output and $\\mathbf{h}_{i, \\ell} = [\\overrightarrow{\\mathbf{h}}_{i,\\ell}; \\overleftarrow{\\mathbf{h}}_{i,\\ell}]$.\nThe weights, $\\mathbf{s}^\\text{task}$, in the linear combination are learned for each end task and normalized by softmax. The scaling factor $\\gamma^\\text{task}$ is used to correct the misalignment between the distribution of biLM hidden states and the distribution of task specific representations.\n$$\nv_i = f(R_i; \\Theta^\\text{task}) = \\gamma^\\text{task} \\sum_{\\ell=0}^L s^\\text{task}_i \\mathbf{h}_{i,\\ell}\n$$\nTo evaluate what kind of information is captured by hidden states across different layers, ELMo is applied on semantic-intensive and syntax-intensive tasks respectively using representations in different layers of biLM:\nSemantic task\n: The\nword sense disambiguation (WSD)\ntask emphasizes the meaning of a word given a context. The biLM top layer is better at this task than the first layer.\nSyntax task\n: The\npart-of-speech\n(POS) tagging\ntask aims to infer the grammatical role of a word in one sentence. A higher accuracy can be achieved by using the biLM first layer than the top layer.\nThe comparison study indicates that syntactic information is better represented at lower layers while semantic information is captured by higher layers. Because different layers tend to carry different type of information,\nstacking them together helps\n.\nUse ELMo in Downstream Tasks\n#\nSimilar to how\nCoVe\ncan help different downstream tasks, ELMo embedding vectors are included in the input or lower levels of task-specific models. Moreover, for some tasks (i.e.,\nSNLI\nand\nSQuAD\n, but not\nSRL\n), adding them into the output level helps too.\nThe improvements brought up by ELMo are largest for tasks with a small supervised dataset. With ELMo, we can also achieve similar performance with much less labeled data.\nSummary\n: The language model pre-training is unsupervised and theoretically the pre-training can be scaled up as much as possible since the unlabeled text corpora are abundant. However, it still has the dependency on task-customized models and thus the improvement is only incremental, while searching for a good model architecture for every task remains non-trivial.\nCross-View Training\n#\nIn ELMo the unsupervised pre-training and task-specific learning happen for two independent models in two separate training stages.\nCross-View Training\n(abbr.\nCVT\n;\nClark et al., 2018\n) combines them into one unified semi-supervised learning procedure where the representation of a biLSTM encoder is improved by both supervised learning with labeled data and unsupervised learning with unlabeled data on auxiliary tasks.\nModel Architecture\n#\nThe model consists of a two-layer bidirectional LSTM encoder and a primary prediction module. During training, the model is fed with labeled and unlabeled data batches alternatively.\nOn\nlabeled examples\n, all the model parameters are updated by standard supervised learning. The loss is the standard cross entropy.\nOn\nunlabeled examples\n, the primary prediction module still can produce a “soft” target, even though we cannot know exactly how accurate they are. In a couple of auxiliary tasks, the predictor only sees and processes a restricted view of the input, such as only using encoder hidden state representation in one direction. The auxiliary task outputs are expected to match the primary prediction target for a full view of input.\nIn this way, the encoder is forced to distill the knowledge of the full context into partial representation. At this stage, the biLSTM encoder is backpropagated but the primary prediction module is\nfixed\n. The loss is to minimize the distance between auxiliary and primary predictions.\nThe overview of semi-supervised language model cross-view training. (Image source:\noriginal paper\n)\nMulti-Task Learning\n#\nWhen training for multiple tasks simultaneously, CVT adds several extra primary prediction models for additional tasks. They all share the same sentence representation encoder.\nDuring supervised training, once one task is randomly selected, parameters in its corresponding predictor and the representation encoder are updated.\nWith unlabeled data samples, the encoder is optimized jointly across all the tasks by minimizing the differences between auxiliary outputs and primary prediction for every task.\nThe multi-task learning encourages better generality of representation and in the meantime produces a nice side-product: all-tasks-labeled examples from unlabeled data. They are precious data labels considering that cross-task labels are useful but fairly rare.\nUse CVT in Downstream Tasks\n#\nTheoretically the primary prediction module can take any form, generic or task-specific design. The examples presented in the CVT paper include both cases.\nIn sequential tagging tasks (classification for every token) like\nNER\nor\nPOS\ntagging, the predictor module contains two fully connected layers and a softmax layer on the output to produce a probability distribution over class labels.\nFor each token $\\mathbf{x}_i$, we take the corresponding hidden states in two layers, $\\mathbf{h}_1^{(i)}$ and $\\mathbf{h}_2^{(i)}$:\n$$\n\\begin{aligned}\np_\\theta(y_i \\mid \\mathbf{x}_i) \n&= \\text{NN}(\\mathbf{h}^{(i)}) \\\\\n&= \\text{NN}([\\mathbf{h}_1^{(i)}; \\mathbf{h}_2^{(i)}]) \\\\\n&= \\text{softmax} \\big( \\mathbf{W}\\cdot\\text{ReLU}(\\mathbf{W'}\\cdot[\\mathbf{h}_1^{(i)}; \\mathbf{h}_2^{(i)}]) + \\mathbf{b} \\big)\n\\end{aligned}\n$$\nThe auxiliary tasks are only fed with forward or backward LSTM state in the first layer. Because they only observe partial context, either on the left or right, they have to learn like a language model, trying to predict the next token given the context. The\nfwd\nand\nbwd\nauxiliary tasks only take one direction. The\nfuture\nand\npast\ntasks take one step further in forward and backward direction, respectively.\n$$\n\\begin{aligned}\np_\\theta^\\text{fwd}(y_i \\mid \\mathbf{x}_i) &= \\text{NN}^\\text{fwd}(\\overrightarrow{\\mathbf{h}}^{(i)}) \\\\\np_\\theta^\\text{bwd}(y_i \\mid \\mathbf{x}_i) &= \\text{NN}^\\text{bwd}(\\overleftarrow{\\mathbf{h}}^{(i)}) \\\\\np_\\theta^\\text{future}(y_i \\mid \\mathbf{x}_i) &= \\text{NN}^\\text{future}(\\overrightarrow{\\mathbf{h}}^{(i-1)}) \\\\\np_\\theta^\\text{past}(y_i \\mid \\mathbf{x}_i) &= \\text{NN}^\\text{past}(\\overleftarrow{\\mathbf{h}}^{(i+1)})\n\\end{aligned}\n$$\nThe sequential tagging task depends on four auxiliary prediction models, their inputs only involving hidden states in one direction: forward, backward, future and past. (Image source:\noriginal paper\n)\nNote that if the primary prediction module has dropout, the dropout layer works as usual when training with labeled data, but it is not applied when generating “soft” target for auxiliary tasks during training with unlabeled data.\nIn the machine translation task, the primary prediction module is replaced with a standard unidirectional LSTM decoder with attention. There are two auxiliary tasks: (1) apply dropout on the attention weight vector by randomly zeroing out some values; (2) predict the future word in the target sequence. The primary prediction for auxiliary tasks to match is the best predicted target sequence produced by running the fixed primary decoder on the input sequence with\nbeam search\n.\nULMFiT\n#\nThe idea of using generative pretrained LM + task-specific fine-tuning was first explored in ULMFiT (\nHoward & Ruder, 2018\n), directly motivated by the success of using ImageNet pre-training for computer vision tasks. The base model is\nAWD-LSTM\n.\nULMFiT follows three steps to achieve good transfer learning results on downstream language classification tasks:\nGeneral LM pre-training\n: on Wikipedia text.\nTarget task LM fine-tuning\n: ULMFiT proposed two training techniques for stabilizing the fine-tuning process. See below.\nDiscriminative fine-tuning\nis motivated by the fact that different layers of LM capture different types of information (see\ndiscussion\nabove). ULMFiT proposed to tune each layer with different learning rates, $\\{\\eta^1, \\dots, \\eta^\\ell, \\dots, \\eta^L\\}$, where $\\eta$ is the base learning rate for the first layer, $\\eta^\\ell$ is for the $\\ell$-th layer and there are $L$ layers in total.\nSlanted triangular learning rates (STLR)\nrefer to a special learning rate scheduling that first linearly increases the learning rate and then linearly decays it. The increase stage is short so that the model can converge to a parameter space suitable for the task fast, while the decay period is long allowing for better fine-tuning.\nTarget task classifier fine-tuning\n: The pretrained LM is augmented with two standard feed-forward layers and a softmax normalization at the end to predict a target label distribution.\nConcat pooling\nextracts max-polling and mean-pooling over the history of hidden states and concatenates them with the final hidden state.\nGradual unfreezing\nhelps to avoid catastrophic forgetting by gradually unfreezing the model layers starting from the last one. First the last layer is unfrozen and fine-tuned for one epoch. Then the next lower layer is unfrozen. This process is repeated until all the layers are tuned.\nThree training stages of ULMFiT. (Image source:\noriginal paper\n)\nGPT\n#\nFollowing the similar idea of ELMo, OpenAI\nGPT\n, short for\nGenerative Pre-training Transformer\n(\nRadford et al., 2018\n), expands the unsupervised language model to a much larger scale by training on a giant collection of free text corpora. Despite of the similarity, GPT has two major differences from ELMo.\nThe model architectures are different: ELMo uses a shallow concatenation of independently trained left-to-right and right-to-left multi-layer LSTMs, while GPT is a multi-layer transformer decoder.\nThe use of contextualized embeddings in downstream tasks are different: ELMo feeds embeddings into models customized for specific tasks as additional features, while GPT fine-tunes the same base model for all end tasks.\nTransformer Decoder as Language Model\n#\nCompared to the\noriginal transformer\narchitecture, the\ntransformer decoder\nmodel discards the encoder part, so there is only one single input sentence rather than two separate source and target sequences.\nThis model applies multiple transformer blocks over the embeddings of input sequences. Each block contains a masked\nmulti-headed self-attention\nlayer and a\npointwise feed-forward\nlayer. The final output produces a distribution over target tokens after softmax normalization.\nThe transformer decoder model architecture in OpenAI GPT.\nThe loss is the negative log-likelihood, same as\nELMo\n, but without backward computation. Let’s say, the context window of the size $k$ is located before the target word and the loss would look like:\n$$\n\\mathcal{L}_\\text{LM} = -\\sum_{i} \\log p(x_i\\mid x_{i-k}, \\dots, x_{i-1})\n$$\nByte Pair Encoding\n#\nByte Pair Encoding\n(\nBPE\n) is used to encode the input sequences. BPE was originally proposed as a data compression algorithm in 1990s and then was adopted to solve the open-vocabulary issue in machine translation, as we can easily run into rare and unknown words when translating into a new language. Motivated by the intuition that rare and unknown words can often be decomposed into multiple subwords, BPE finds the best word segmentation by iteratively and greedily merging frequent pairs of characters.\nSupervised Fine-Tuning\n#\nThe most substantial upgrade that OpenAI GPT proposed is to get rid of the task-specific model and use the pre-trained language model directly!\nLet’s take classification as an example. Say, in the labeled dataset, each input has $n$ tokens, $\\mathbf{x} = (x_1, \\dots, x_n)$, and one label $y$. GPT first processes the input sequence $\\mathbf{x}$ through the pre-trained transformer decoder and the last layer output for the last token $x_n$ is $\\mathbf{h}_L^{(n)}$. Then with only one new trainable weight matrix $\\mathbf{W}_y$, it can predict a distribution over class labels.\n$$\nP(y\\mid x_1, \\dots, x_n) = \\text{softmax}(\\mathbf{h}_L^{(n)}\\mathbf{W}_y)\n$$\nThe loss is to minimize the negative log-likelihood for true labels. In addition, adding the LM loss as an auxiliary loss is found to be beneficial, because:\n(1) it helps accelerate convergence during training and\n(2) it is expected to improve the generalization of the supervised model.\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{cls} &= \\sum_{(\\mathbf{x}, y) \\in \\mathcal{D}} \\log P(y\\mid x_1, \\dots, x_n) = \\sum_{(\\mathbf{x}, y) \\in \\mathcal{D}} \\log \\text{softmax}(\\mathbf{h}_L^{(n)}(\\mathbf{x})\\mathbf{W}_y) \\\\\n\\mathcal{L}_\\text{LM} &= -\\sum_{i} \\log p(x_i\\mid x_{i-k}, \\dots, x_{i-1}) \\\\\n\\mathcal{L} &= \\mathcal{L}_\\text{cls} + \\lambda \\mathcal{L}_\\text{LM}\n\\end{aligned}\n$$\nWith similar designs, no customized model structure is needed for other end tasks (see Fig. 7). If the task input contains multiple sentences, a special delimiter token (\n$\n) is added between each pair of sentences. The embedding for this delimiter token is a new parameter we need to learn, but it should be pretty minimal.\nFor the sentence similarity task, because the ordering does not matter, both orderings are included. For the multiple choice task, the context is paired with every answer candidate.\nTraining objects in slightly modified GPT transformer models for downstream tasks. (Image source:\noriginal paper\n)\nSummary\n: It is super neat and encouraging to see that such a general framework is capable to beat SOTA on most language tasks at that time (June 2018). At the first stage, generative pre-training of a language model can absorb as much free text as possible. Then at the second stage, the model is fine-tuned on specific tasks with a small labeled dataset and a minimal set of new parameters to learn.\nOne limitation of GPT is its uni-directional nature — the model is only trained to predict the future left-to-right context.\nBERT\n#\nBERT\n, short for\nBidirectional Encoder Representations from Transformers\n(\nDevlin, et al., 2019\n) is a direct descendant to\nGPT\n: train a large language model on free text and then fine-tune on specific tasks without customized network architectures.\nCompared to GPT, the largest difference and improvement of BERT is to make training\nbi-directional\n. The model learns to predict both context on the left and right. The paper according to the ablation study claimed that:\n“bidirectional nature of our model is the single most important new contribution”\nPre-training Tasks\n#\nThe model architecture of BERT is a multi-layer bidirectional Transformer encoder.\nRecap of Transformer Encoder model architecture. (Image source:\nTransformer paper\n)\nTo encourage the bi-directional prediction and sentence-level understanding, BERT is trained with two tasks instead of the basic language task (that is, to predict the next token given context).\n*\nTask 1: Mask language model (MLM)\nFrom\nWikipedia\n: “A cloze test (also cloze deletion test) is an exercise, test, or assessment consisting of a portion of language with certain items, words, or signs removed (cloze text), where the participant is asked to replace the missing language item. … The exercise was first described by W.L. Taylor in 1953.”\nIt is unsurprising to believe that a representation that learns the context around a word rather than just after the word is able to better capture its meaning, both syntactically and semantically. BERT encourages the model to do so by training on the\n“mask language model” task\n:\nRandomly mask 15% of tokens in each sequence. Because if we only replace masked tokens with a special placeholder\n[MASK]\n, the special token would never be encountered during fine-tuning. Hence, BERT employed several heuristic tricks:\n(a) with 80% probability, replace the chosen words with\n[MASK]\n;\n(b) with 10% probability, replace with a random word;\n(c) with 10% probability, keep it the same.\nThe model only predicts the missing words, but it has no information on which words have been replaced or which words should be predicted. The output size is only 15% of the input size.\nTask 2: Next sentence prediction\nMotivated by the fact that many downstream tasks involve the understanding of relationships between sentences (i.e.,\nQA\n,\nNLI\n), BERT added another auxiliary task on training a\nbinary classifier\nfor telling whether one sentence is the next sentence of the other:\nSample sentence pairs (A, B) so that:\n(a) 50% of the time, B follows A;\n(b) 50% of the time, B does not follow A.\nThe model processes both sentences and output a binary label indicating whether B is the next sentence of A.\nThe training data for both auxiliary tasks above can be trivially generated from any monolingual corpus. Hence the scale of training is unbounded. The training loss is the sum of the mean masked LM likelihood and mean next sentence prediction likelihood.\nComparison of BERT, OpenAI GPT and ELMo model architectures. (Image source:\noriginal paper\n)\nInput Embedding\n#\nThe input embedding is the sum of three parts:\nWordPiece tokenization embeddings\n: The\nWordPiece\nmodel\nwas originally proposed for Japanese or Korean segmentation problem. Instead of using naturally split English word, they can be further divided into smaller sub-word units so that it is more effective to handle rare or unknown words. Please read\nlinked\npapers\nfor the optimal way to split words if interested.\nSegment embeddings\n: If the input contains two sentences, they have sentence A embeddings and sentence B embeddings respectively and they are separated by a special character\n[SEP]\n; Only sentence A embeddings are used if the input only contains one sentence.\nPosition embeddings\n: Positional embeddings are learned rather than hard-coded.\nBERT input representation. (Image source:\noriginal paper\n)\nNote that the first token is always forced to be\n[CLS]\n— a placeholder that will be used later for prediction in downstream tasks.\nUse BERT in Downstream Tasks\n#\nBERT fine-tuning requires only a few new parameters added, just like OpenAI GPT.\nFor classification tasks, we get the prediction by taking the final hidden state of the special first token\n[CLS]\n, $\\mathbf{h}^\\text{[CLS]}_L$, and multiplying it with a small weight matrix, $\\text{softmax}(\\mathbf{h}^\\text{[CLS]}_L \\mathbf{W}_\\text{cls})$.\nFor\nQA\ntasks like SQuAD, we need to predict the text span in the given paragraph for an given question. BERT predicts two probability distributions of every token, being the start and the end of the text span. Only two new small matrices, $\\mathbf{W}_\\text{s}$ and $\\mathbf{W}_\\text{e}$, are newly learned during fine-tuning and $\\text{softmax}(\\mathbf{h}^\\text{(i)}_L \\mathbf{W}_\\text{s})$ and $\\text{softmax}(\\mathbf{h}^\\text{(i)}_L \\mathbf{W}_\\text{e})$ define two probability distributions.\nOverall the add-on part for end task fine-tuning is very minimal — one or two weight matrices to convert the Transform hidden states to an interpretable format. Check the paper for implementation details for other cases.\nTraining objects in slightly modified BERT models for downstream tasks. (Image source:\noriginal paper\n)\nA summary table compares differences between fine-tuning of\nOpenAI GPT\nand BERT.\n| |\nOpenAI GPT\n|\nBERT\n|\n| Special char |\n[SEP]\nand\n[CLS]\nare only introduced at fine-tuning stage. |\n[SEP]\nand\n[CLS]\nand sentence A/B embeddings are learned at the pre-training stage. |\n| Training process | 1M steps, batch size 32k words. | 1M steps, batch size 128k words. |\n| Fine-tuning | lr = 5e-5 for all fine-tuning tasks. | Use task-specific lr for fine-tuning. |\nALBERT\n#\nALBERT\n(\nLan, et al. 2019\n), short for\nA Lite BERT\n, is a light-weighted version of\nBERT\nmodel. An ALBERT model can be trained 1.7x faster with 18x fewer parameters, compared to a BERT model of similar configuration. ALBERT incorporates three changes as follows: the first two help reduce parameters and memory consumption and hence speed up the training speed, while the third one proposes a more chanllenging training task to replace the next sentence prediction (NSP) objective.\nFactorized Embedding Parameterization\n#\nIn BERT, the WordPiece tokenization embedding size $E$ is configured to be the same as the hidden state size $H$. That is saying, if we want to increase the model size (larger $H$), we need to learn a larger tokenization embedding too, which is expensive because it depends on the vocabulary size ($V$).\nConceptually, because the tokenization embedding is expected to learn\ncontext-independent\nrepresentation and the hidden states are\ncontext-dependent\n, it makes sense to separate the size of the hidden layers from the size of vocabulary embedding. Using factorized embedding parameterization, the large vocabulary embedding matrix of size $V \\times H$ is decomposed into two small matrices of size $V \\times E$ and $E \\times H$. Given $H \\gt E$ or even $H \\gg E$, factorization can result in significant parameter reduction.\nCross-layer Parameter Sharing\n#\nParameter sharing across layers can happen in many ways: (a) only share feed-forward part; (b) only share attention parameters; or (c) share all the parameters. This technique reduces the number of parameters by a ton and does not damage the performance too much.\nSentence-Order Prediction (SOP)\n#\nInterestingly, the\nnext sentence prediction (NSP)\ntask of BERT turned out to be too easy. ALBERT instead adopted a sentence-order prediction (SOP)\nself-supervised\nloss,\nPositive sample: two consecutive segments from the same document.\nNegative sample: same as above, but the segment order is switched.\nFor the NSP task, the model can make reasonable predictions if it is able to detect topics when A and B are from different contexts. In comparison, SOP is harder as it requires the model to fully understand the coherence and ordering between segments.\nGPT-2\n#\nThe\nOpenAI\nGPT-2\nlanguage model is a direct successor to\nGPT\n. GPT-2 has 1.5B parameters, 10x more than the original GPT, and it achieves SOTA results on 7 out of 8 tested language modeling datasets in a\nzero-shot transfer setting\nwithout any task-specific fine-tuning. The pre-training dataset contains 8 million Web pages collected by crawling qualified outbound links from\nReddit\n. Large improvements by OpenAI GPT-2 are specially noticeable on small datasets and datasets used for measuring\nlong-term dependency\n.\nZero-Shot Transfer\n#\nThe pre-training task for GPT-2 is solely language modeling. All the downstream language tasks are framed as predicting conditional probabilities and there is no task-specific fine-tuning.\nText generation is straightforward using LM.\nMachine translation task, for example, English to Chinese, is induced by conditioning LM on pairs of “English sentence = Chinese sentence” and “the target English sentence =” at the end.\nFor example, the conditional probability to predict might look like:\nP(? | I like green apples. = 我喜欢绿苹果。 A cat meows at him. = 一只猫对他喵。It is raining cats and dogs. =\")\nQA task is formatted similar to translation with pairs of questions and answers in the context.\nSummarization task is induced by adding\nTL;DR:\nafter the articles in the context.\nBPE on Byte Sequences\n#\nSame as the original GPT, GPT-2 uses\nBPE\nbut on\nUTF-8\nbyte sequences. Each byte can represent 256 different values in 8 bits, while UTF-8 can use up to 4 bytes for one character, supporting up to $2^{31}$ characters in total. Therefore, with byte sequence representation we only need a vocabulary of size 256 and do not need to worry about pre-processing, tokenization, etc. Despite of the benefit, current byte-level LMs still have non-negligible performance gap with the SOTA word-level LMs.\nBPE merges frequently co-occurred byte pairs in a greedy manner. To prevent it from generating multiple versions of common words (i.e.\ndog.\n,\ndog!\nand\ndog?\nfor the word\ndog\n), GPT-2 prevents BPE from merging characters across categories (thus\ndog\nwould not be merged with punctuations like\n.\n,\n!\nand\n?\n). This tricks help increase the quality of the final byte segmentation.\nUsing the byte sequence representation, GPT-2 is able to assign a probability to any Unicode string, regardless of any pre-processing steps.\nModel Modifications\n#\nCompared to GPT, other than having many more transformer layers and parameters, GPT-2 incorporates only a few architecture modifications:\nLayer normalization\nwas moved to the input of each sub-block, similar to a residual unit of type\n“building block”\n(differently from the original type\n“bottleneck”\n, it has batch normalization applied before weight layers).\nAn additional layer normalization was added after the final self-attention block.\nA modified initialization was constructed as a function of the model depth.\nThe weights of residual layers were initially scaled by a factor of $1/ \\sqrt{N}$ where N is the number of residual layers.\nUse larger vocabulary size and context size.\nRoBERTa\n#\nRoBERTa\n(short for\nR\nobustly\no\nptimized\nBERT\na\npproach;\nLiu, et al. 2019\n) refers to a new receipt for training BERT to achieve better results, as they found that the original BERT model is significantly undertrained. The receipt contains the following learnings:\nTrain for longer with bigger batch size.\nRemove the\nnext sentence prediction (NSP)\ntask.\nUse longer sequences in training data format. The paper found that using individual sentences as inputs hurts downstream performance. Instead we should use multiple sentences sampled contiguously to form longer segments.\nChange the masking pattern dynamically. The original BERT applies masking once during the data preprocessing stage, resulting in a static mask across training epochs. RoBERTa applies masks in 10 different ways across 40 epochs.\nRoBERTa also added a new dataset\nCommonCrawl News\nand further confirmed that pretraining with\nmore data helps\nimprove the performance on downstream tasks. It was trained with the\nBPE on byte sequences\n, same as in\nGPT-2\n. They also found that choices of hyperparameters have a big impact on the model performance.\nT5\n#\nThe language model\nT5\nis short for\n“Text-to-Text Transfer Transformer”\n(\nRaffel et al., 2020\n). The encoder-decoder implementation follows the\noriginal Transformer\narchitecture: tokens → embedding → encoder → decoder → output. T5 adopts the framework “Natural Language Decathlon” (\nMcCann et al., 2018\n), where many common NLP tasks are translated into question-answering over a context. Instead of an explicit QA format, T5 uses short task prefixes to distinguish task intentions and separately fine-tunes the model on every individual task. The text-to-text framework enables easier transfer learning evaluation with the same model on a diverse set of tasks.\nA diagram of T5 task evaluation. The text-to-text framework casts every task into a generic form: feeding input text to predict some target text. (Image source:\nRaffel et al., 2020\n)\nThe model is trained on Web corpus extracted from Apr 2019 with various filters applied. The model is fine-tuned for each downstream task separately via “adapter layers” (add an extra layer for training) or “gradual unfreezing” (see\nULMFiT\n). Both fine-tuning approaches only update partial parameters while keeping the majority of the model parameters unchanged. T5-11B achieved SOTA results on many NLP tasks.\nAs the authors mentioned in the paper “…our goal is not to propose new methods but instead to provide a comprehensive perspective on where the field stands”, the T5 long paper described a lot of training setup and evaluation processes in detail, a good read for people who are interested in training a LM from scratch.\nGPT-3\n#\nGPT-3\n(\nBrown et al., 2020\n) has the same architecture as\nGPT-2\nbut contains 175B parameters, 10x larger than GPT-2 (1.5B). In addition, GPT-3 uses alternating dense and locally banded sparse attention patterns, same as in\nsparse transformer\n. In order to fit such a huge model across multiple GPUs, GPT-3 is trained with partitions along both width and depth dimension. The training data is a filtered version of Common Crawl mixed with a few other high-quality curated datasets. To avoid the contamination that downstream tasks might appear in the training data, the authors attempted to remove all the overlaps with all the studied benchmark dataset from the training dataset. Unfortunately the filtering process is not perfect due to a bug.\nTraining datasets for GPT-3. Note that the occurrence of each dataset during training is not proportional to the dataset size. (Table source:\nBrown et al., 2020\n)\nFor all the downstream evaluation, GPT-3 is tested in the few-shot setting without any gradient-based fine-tuning. Here the few-shot examples are provided as part of the prompt. GPT-3 achieves strong performance on many NLP datasets, comparable with fine-tuned BERT models.\nThe evaluation performance increases with the model size and the number of examples. (Image source:\nBrown et al., 2020\n)\nXLNet\n#\nThe\nAutoregressive (AR)\nmodel such as GPT and\nautoencoder (AE)\nmodel such as BERT are two most common ways for language modeling. However, each has their own disadvantages: AR does not learn the bidirectional context, which is needed by downstream tasks like reading comprehension and AE assumes masked positions are independent given all other unmasked tokens which oversimplifies the long context dependency.\nXLNet\n(\nYang et al. 2019\n) generalizes the AE method to incorporate the benefits of AR. XLNet proposed the\npermutation language modeling\nobjective. For a text sequence, it samples a factorization order $\\mathbf{z}$ and decomposes the likelihood $p_\\theta(\\mathbf{x})$ according to this factorization order,\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{XLNet} \n&= - \\mathbb{E}_{\\mathbf{z} \\sim \\mathcal{Z}_T} \\Big[ \\sum_{t=1}^T \\log p_\\theta (X_{z_t} = x \\mid \\mathbf{x}_{\\mathbf{z}_{<{t}}})\\Big] \\\\\n&= - \\mathbb{E}_{\\mathbf{z} \\sim \\mathcal{Z}_T} \\Big[ \\log \\frac{ \\exp(e(x)^\\top \\color{red}{h_\\theta (\\mathbf{x}_{\\mathbf{z}_{<{t}}})}) }{ \\sum_{x'} \\exp(e(x')^\\top \\color{red}{h_\\theta (\\mathbf{x}_{\\mathbf{z}_{<{t}}})}) } \\Big] \\\\\n&= - \\mathbb{E}_{\\mathbf{z} \\sim \\mathcal{Z}_T} \\Big[ \\log \\frac{ \\exp(e(x)^\\top \\color{blue}{g_\\theta (\\mathbf{x}_{\\mathbf{z}_{<{t}}}, z_t)}) }{ \\sum_{x'} \\exp(e(x')^\\top \\color{blue}{g_\\theta (\\mathbf{x}_{\\mathbf{z}_{<{t}}}, z_t)}) } \\Big]\n\\end{aligned}\n$$\nwhere $\\mathcal{Z}_T$ is a set of all possible permutation of length $T$; $z_t$ and $\\mathbf{z}_{