{ "source_file": "0930", "source_platform": "个人博客", "author": "Lilian Weng / Lil'Log", "title_listed": "Learning with not Enough Data Part 3: Data Generation", "keywords": "数据增强,合成数据,生成", "domain": "数据处理", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/", "publish_date": "2022-04", "corpus_stage": "数据处理", "notes": "数据生成系列综述之三", "id": "1cc4df7eb2c7", "collected_at": "2026-10-02T14:13:06", "title": "Learning with not Enough Data Part 3: Data Generation | Lil'Log", "body_markdown": "Here comes the Part 3 on learning with not enough data (Previous:\nPart 1\nand\nPart 2\n). Let’s consider two approaches for generating synthetic data for training.\nAugmented data\n. Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a\nprevious post\non contrastive learning. For the sake of post completeness, I\nduplicate\nthe section on data augmentation here with some edits.\nNew data\n. Given few or even no data points, we can rely on powerful pretrained models to generate a number of\nnew\ndata points. This is especially true in recent years given the fast progress in large pretrained\nlanguage models (LM)\n. Few shot prompting is shown to be effective for LM to learn within context without extra training.\nData Augmentation\n#\nThe goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.\nImage Augmentation\n#\nBasic Image Processing Operations\n#\nThere are several ways to modify an image while retaining its semantic information. We can use any one of the following augmentation or a composition of multiple operations.\nRandom cropping and then resize back to the original size.\nRandom color distortions\nRandom Gaussian blur\nRandom color jittering\nRandom horizontal flip\nRandom grayscale conversion\nAnd many more. Check\nPIL.ImageOps\nfor inspiration.\nTask-Specific Augmentation Strategies\n#\nIf the downstream task is known, it is possible to learn the optimal augmentation strategies (i.e. what processing operations to use and how to combine them in sequence) to maximize the downstream task performance.\nAutoAugment\n(\nCubuk, et al. 2018\n) is inspired by\nneural architecture search\n, AutoAugment frames the problem of learning best data augmentation operations (i.e. shearing, rotation, invert, etc.) for image classification as an RL problem and looks for the combination that leads to the highest accuracy on the evaluation set. AutoAugment can be executed in adversarial fashion (\nZhang, et al 2019\n).\nRandAugment\n(\nCubuk et al., 2019\n) greatly reduces the search space of AutoAugment by controlling the magnitudes of different transformation operations with a single magnitude parameter.\nPopulation based augmentation\n(PBA;\nHo et al., 2019\n) combines PBT (“population based training”;\nJaderberg et al, 2017\n) with AutoAugment, using the evolutionary algorithm to train a population of children models in parallel to evolve the best augmentation strategies.\nUnsupervised Data Augmentation\n(UDA;\nXie et al., 2019\n), among a set of possible augmentation strategies, selects a subset to minimize the KL divergence between the predicted distribution over an unlabelled example and its unlabelled augmented version.\nImage Mixture\n#\nImage mixture methods can construct new training examples from existing data points.\nMixup\n(\nZhang et al., 2018\n) runs global-level mixture by creating a weighted pixel-wise combination of two existing images $I_1$ and $I_2$: $I_\\text{mixup} \\gets \\alpha I_1 + (1-\\alpha) I_2$ and $\\alpha \\in [0, 1]$.\nCutmix\n(\nYun et al., 2019\n) does region-level mixture by generating a new example by combining a local region of one image with the rest of the other image. $I_\\text{cutmix} \\gets \\mathbf{M}_b \\odot I_1 + (1-\\mathbf{M}_b) \\odot I_2$, where $\\mathbf{M}_b \\in \\{0, 1\\}^I$ is a binary mask and $\\odot$ is element-wise multiplication. It is equivalent to filling the\ncutout\n(\nDeVries & Taylor 2017\n) region with the same region from another image.\nGiven a query $\\mathbf{q}$,\nMoCHi\n(“mixing of contrastive hard negatives”;\nKalantidis et al. 2020\n) maintains a queue of $K$ negative features $Q={\\mathbf{n}_1, \\dots, \\mathbf{n}_K }$ and sorts these negative features by similarity to the query, $\\mathbf{q}^\\top \\mathbf{n}$, in descending order. The first $N$ items in the queue are considered as the hardest negatives, $Q^N$. Then synthetic hard examples can be generated by $\\mathbf{h} = \\tilde{\\mathbf{h}} / |\\tilde{\\mathbf{h}}|_2$ where $\\tilde{\\mathbf{h}} = \\alpha\\mathbf{n}_i + (1-\\alpha) \\mathbf{n}_j$ and $\\alpha \\in (0, 1)$. Even harder examples can be created by mixing with the query feature, $\\mathbf{h}’ = \\tilde{\\mathbf{h}’} / |\\tilde{\\mathbf{h}’}|_2$ where $\\tilde{\\mathbf{h}’} = \\beta\\mathbf{q} + (1-\\beta) \\mathbf{n}_j$ and $\\beta \\in (0, 0.5)$.\nText Augmentation\n#\nLexical Edits\n#\nEasy Data Augmentation\n(EDA;\nWei & Zou 2019\n) defines a set of simple but powerful operations for text augmentation. Given a sentence, EDA randomly chooses and applies one of four simple operations:\nSynonym replacement (SR): Replace $n$ random non-stop words with their synonyms.\nRandom insertion (RI): Place a random synonym of a randomly selected non-stop word in the sentence at a random position.\nRandom swap (RS): Randomly swap two words and repeat $n$ times.\nRandom deletion (RD): Randomly delete each word in the sentence with probability $p$.\nwhere $p=\\alpha$ and $n=\\alpha \\times \\text{sentence_length}$, with the intuition that longer sentences can absorb more noise while maintaining the original label. The hyperparameter $\\alpha$ roughly indicates the percent of words in one sentence that may be changed by one augmentation.\nEDA is shown to improve the classification accuracy on several classification benchmark datasets compared to baseline without EDA. The performance lift is more significant on a\nsmaller\ntraining set. All the four operations in EDA help improve the classification accuracy, but get to optimal at different $\\alpha$’s.\nEDA leads to performance improvement on several classification benchmarks. (Image source:\nWei & Zou 2019\n)\nContextual\nAugmentation (\nKobayashi, 2018\n) replaces word $w_i$ at position $i$ by sampling from a probability distribution learned by a bidirectional LM such as BERT, $p(.\\mid S\\setminus{w_i})$. In this way, the words are substituted by synonyms, or similar words suitable for the context. To guarantee such operations do not alter the labels, the LM is fit to be label-conditioned bidirectional LM. Conditional BERT (CBERT;\nXing Wu et al. 2018\n) extends BERT to predict masked tokens conditioned on the class label and can be used for contextual augmentation prediction.\nBack-translation\n#\nBack-translation\nproduces augmented data by translating text samples to another language and then translating them back. The translation happens in two ways and both directions should have decent enough performance to avoid significant loss of semantic meaning.\nMix-up\n#\nIt is also possible to apply\nMixup\nto text (\nGuo et al. 2019\n) but on the embedding space to obtain some performance gain. The proposed method relies on a specially designed model architecture to operate the prediction on the word or sentence embedding. Adding adversarial noise in the embedding space as a way of data augmentation is shown to improve the generalization of model training (\nZhu et al. 2019\n).\nAudio Augmentation\n#\nHere is a list of several commonly used audio data augmentation methods, operated on raw audio or spectrograms, summarized by\nWang & van den Oord (2021)\n.\nAudio mixup.\nGiven two audio clips $\\mathbf{x}_1$ and $\\mathbf{x}_2$, the mixed-up version $\\hat{\\mathbf{x}} = \\alpha \\mathbf{x}_1 + (1-\\alpha)\\mathbf{x}_2$ should be associated with the label of the more dominant input. The audio mixup augments the data with more realistic noise.\nTime masking.\nA small consecutive chunk of the audio can be masked without losing semantic information.\nFrequency masking.\nA small amount of frequency components on the spectrogram can be dropped off and it should not change the associated label.\nFrequency shift.\nThe spectrogram can be shifted by an integer between $[-F, F]$, where $F$ is the maximum shift size. It is a cheap augmentation to change the pitch of the audio.\nArchitectural Augmentation\n#\nModels with\ndropout\nlayers can create augmented samples by applying different dropout masks on the same input sample. For example, in the contrastive learning model\nSimCSE\n(\nGuo et al. 2021\n), a sample is simply fed into the encoder twice with different dropout masks and these two versions are the positive pair where the other in-batch samples are considered as negative pairs.\nDropout augments data by adding noise onto the internal representation of the model. It can be applied in a more structured way, such as in\ncutoff\n(\nShen et al. (2020)\n), where random chunks of the token embedding matrix are removed.\nData Synthesis\n#\nGiven that generating high-quality, photorealistic images is a lot more difficult than generating human-like natural language text and recent success with large pretrained language models, this section only focuses on text generation. To read more on how to synthesize realistic images, check posts on\nGAN\n,\nVAE\n,\nflow\nand\ndiffusion\nmodels.\nLanguage Model as Noisy Annotator\n#\nWang et al. (2021)\nexplored ways to leverage GPT-3 as a weak annotator via few-shot prompting, achieving 10x cheaper than human labeling. The paper argues that by using data labeled by GPT-3, it essentially performs\nself-training\n: The predictions on unlabeled samples apply entropy regularization on the model to avoid high class overlaps so as to help improve the model performance.\nIllustration of how to use GPT-3 to generate more training data with the human-in-the-loop active learning pipeline to improve the data quality. (Image source:\nWang et al. 2021\n)\nGPT-3-labeled samples selected by\nactive learning\nwith highest uncertainty are sent to human labelers to be re-annotated. The few-shot prompt contains a small number of human labeled examples and thus the labeling cost is restricted. Synthetic samples are ranked by predicted logits of label $y$ and those with the lowest scores go through relabeling.\nGPT-3 labeling achieves better results in the low-cost regime, but has a gap with human labeling when enough money is spent on data collection. This implies the following inequation, although to what extent “a lot” or “noisy” means depends on the task details.\nA lot of high-quality data > A lot of noisy data > A little high quality data\n.\nGPT-3 labeling technique improves the classification performance in the low-cost regime. (Image source:\nWang et al. 2021\n)\nLanguage Model as Data Generator\n#\nIf enough training dataset for text classification tasks are available, we can fine-tune language models to synthesize more training samples conditioned on labels (\nAnaby-Tavor et al. 2019\n,\nKumar et al. 2021\n).\nLanguage-model-based data augmentation\n(\nLAMBADA\n;\nAnaby-Tavor et al. 2019\n) takes such an idea, where the process involves fine-tuning both a classifier and a sample generation model.\nTrain a baseline classifier using the existing training dataset: $h = \\mathcal{A}(\\mathcal{D}_\\text{train})$.\nIndependently of step 1, a LM $\\mathcal{M}$ is fine-tuned on $\\mathcal{D}_{\\text{train}}$ to obtain $\\mathcal{M}_{\\text{tuned}}$.\nSynthesize a labeled dataset $\\mathcal{D}^*$ by generating the continuation of the sequence\ny[SEP]\nuntil\nEOS\nusing $\\mathcal{M}_\\text{tuned}$.\nFilter synthesized dataset by,\n(1) Verifying that the predicted label is correct $h(x)=y$;\n(2) Selecting the top ranked samples when they are ranked by the classifier probability. $\\mathcal{D}_\\text{syn} \\subset \\mathcal{D}^*$. They generate 10x more samples needed for augmentation and only the top 10% synthesized samples with highest confidence scores remain.\nThe final classifier is trained on $\\mathcal{D}_\\text{syn} \\cup \\mathcal{D}_\\text{train}$ . The process can be repeated multiple times, but it is unclear whether the benefit would quickly diminish or the repetitive process would bring in self-bias.\nAccuracy of LAMBADA vs. other generative approaches over all datasets and classifiers. (Image source:\nAnaby-Tavor et al. 2019\n)\nTo simplify LAMBADA, we can actually remove the dependency of a fine-tuned generation model and an existing training dataset of a decent size (\nStep 2\nabove).\nUnsupervised data generation\n(\nUDG\n;\nWang et al. 2021\n) relies on few-shot prompting on a large pretrained language model to generate high-quality synthetic data for training. Opposite to the above approach where LM is asked to predict $y$ given $\\mathbf{x}$, UDG instead synthetizes the inputs $\\mathbf{x}$ given labels $y$. Then a task-specific model is trained on this synthetic dataset.\nSchick & Schutze (2021)\nproposed a similar idea but on the NLI task instead of classification, asking PLM to write sentence pairs that are similar or different while the model is prompted with task-specific instructions.\nIllustration of the unsupervised data generation (UDG) framework. (Image source:\nWang et al., 2021\n)\nThe few-shot prompts of UDG contain a small number of unlabeled examples, as well as a task-specific natural language description of the desired label. Because some generated examples are noisy, they implemented\nnoisy label annealing\n(\nNLA\n) techniques to filter potentially misaligned samples out during the training processes. NLA gradually removes noisy training signals in time during training when the model starts to disagree with its pseudo label with high confidence. At each training step $t$, a given example $(\\mathbf{x}_i, \\hat{y}_i)$ is considered noisy and should be removed if:\nThe model predicted probability is higher than a threshold $p(\\bar{y}_i \\vert \\mathbf{x}_i) > \\mu_t$ where $\\bar{y}_i = \\arg\\max_y p(y \\vert \\mathbf{x}_i)$;\nAnd the predicted label is different from the synthetic label, $\\bar{y}_i \\neq \\hat{y}_i$.\nNote that the threshold $\\mu_t$ is time-dependent, initialized as 0.9 and then gradually annealed to $1/\\text{num_of_classes}$ in time.\nAs shown in their experiments, the improvement of UDG over few-shot inference is quit significant, where NLA brings in some extra boost. The results are even comparable with supervised fine-tuning on several cases.\nComparison of accuracy of UDG and other methods on different classification datasets. (Image source:\nWang et al., 2021\n)\nHan et al (2021)\nachieved SOTA results on translation tasks using few-shot data generation, distillation and back-translation. The proposed method contains the following steps, assuming no access to paired translation data:\nZero-shot Generation.\nFirst use the zero-shot translation ability of a pre-trained LM to generate translations for a small set of unlabeled sentences.\nFew-shot Generation.\nThen amplify these zero-shot translations by using them as few-shot demonstrations to gather an even larger synthetic dataset.\nDistillation.\nFine-tune the model on this dataset. The translation task is formulated as a language modeling task\n[L1] [[TRANSLATE]] [L2] .\ngiven a pair of two sequences\n\nin two different languages. At test-time, the LM is prompted with\n[L1] [[TRANSLATE]] [L2]\nand a candidate translation\n\nis parsed from the sampled completion.\nBack-translation.\nContinue fine-tuning on the back-translation dataset where the order of samples is reversed,\n\n.\nStep 1-4 can be repeated.\nAlgorithm of using distillation and back-translation to train a language model on translation tasks. (Image source:\nHan et al. 2021\n)\nThe success of the above method depends on a good pretrained LM to kick off the initial translation dataset. Iterative few-shot generation and distillation with back-translation is an effective way to extract and refine the translation capability out of a pretrained LM and further to distill that into a new model.\nComparison of BLEU scores of the translation models of different training runs using: only distillation, back-translation, both and with more monolingual training data. (Image source:\nHan et al. 2021\n)\nHow to Quantify Generated Data Quality?\n#\nGiven all the generated data, either by data augmentation or data synthesis, how can we quantify data quality in terms of how they improve model generalization?\nGontijo-Lopes et al. (2020)\nintroduced two dimensions to track, affinity and diversity.\nAffinity\nis a model-sensitive metric for\ndistribution shift\n, quantifying how much an augmentation shifts the training data distribution from what a model learned.\nDefinition: The performance difference between the model tested on clean data vs augmented data, while the model is trained on clean data.\nAs a comparison, KL can also measure distribution shift but does not consider the model performance.\nDiversity\nis a measure of\naugmentation complexity\n, measuring the complexity of the augmented data with respect to the model and learning procedure.\nDefinition: The final training loss of a model trained with a given augmentation.\nAnother potential diversity measure is the entropy of the transformed data.\nA third potential diversity measure is the training time needed for a model to reach a given training accuracy threshold.\nAll three metrics above are correlated.\nThe final model performance is dependent on both metrics to be high enough.\n(a) Left: A scatter plot of affinity vs diversity metric, where each point represents a different augmentation method and its color indicates the final test accuracy. (b) Right: The conceptual illustration of the relationship between clean and augmented data in different regions of affinity and diversity metrics. (Image source:\nGontijo-Lopes et al. 2020\n)\nThere are many quantitative metrics on relevancy and diversity, in different formations depending on whether a reference is available, such as perplexity, BLEU for text and inception score for images. I’m skipping the list of concrete quantitative metrics on quality here, given it could be very long.\nTraining with Noisy Data\n#\nIt is convenient to collect a large amount of noisy data via model generation or data augmentation, but it is hard to guarantee that augmented and generated data can be 100% accurate. Knowing that deep neural networks can easily overfit noisy labels and “memotize” corrupted labels, we can apply the techniques for training on noisy labels (\nnoise-robust training\n) when using generated data to stabilize and optimize the performance. Please check this\nsurvey paper (Song et al. 2021)\non learning from noisy labels for a more thorough coverage of related work.\nRegularization and Robust Architecture\n#\nGenerally speaking, mechanisms designed for avoiding overfitting should help improve training robustness when working with moderately noisy data, such as weight decay, dropout, batch normalization. In fact, good data augmentation (i.e. only non-essential attributes are modified) can be considered as a way of regularization as well.\nA different approach is to enhance the network with a dedicated\nnoisy adaptation layer\nto approximate the unknown projection of label corruption (\nSukhbaatar et al. 2015\n,\nGoldberger & Ben-Reuven, 2017\n).\nSukhbaatar et al. (2015)\nintroduced an extra linear layer $Q$ into the network architecture to adapt the predictions to match the noisy label distribution. The noise matrix $Q$ is initially\nfixed\nto the identity function while only the base model parameters is updated. After some time, $Q$ starts to be updated and expected to capture the noise in the data. The noise matrix is trained with regularization to encourage it to match the noise distribution while keeping the base model prediction accurate for true labels.\n(a) Left: A noise matrix $Q$ is added between softmax and the final output for the loss. (b) Right: The noise matrix $Q$ is fixed at the identity function initially and only gets updated with regularization after some training. (Image source:\nSukhbaatar et al. 2015\n)\nHowever, it is hard to guarantee such a noise matrix layer would only capture the noise transition distribution and it is actually non-trivial to learn.\nGoldberger & Ben-Reuven (2017)\n) proposed to add an additional softmax layer end-to-end with the base model and apply the\nEM algorithm\nby treating the correct labels as latent random variable and the noise processes as a communication channel with unknown parameters.\nRobust Learning Objective\n#\nBesides the most commonly used cross entropy loss, some other choices of learning objectives are shown to be more robust to noisy labels.\nFor example,\nMAE\n(mean absolute error) is more robust to noisy labels than CCE (categorical cross entropy), as it treats every sample equally (\nGhosh et al. 2017\n). Lack of different weighting among training samples of MAE lead to significantly longer training time. Motivated by the tradeoff between MAE and CCE,\nZhang & Sabuncu (2018)\nproposed\ngeneralized cross entropy\n(\nGCE\n), a generalization of CCE loss to be robust to noisy data.\nTo exploit the benefits of both the noise-robustness provided by MAE and the implicit weighting scheme of CCE, GCE adopts the the negative Box-Cox transformation as a loss function:\n$$\n\\mathcal{L}_q(f(\\mathbf{x}_i, y_i = j)) = \\frac{1 - f^{(j)}(\\mathbf{x}_i)^q}{q}\n$$\nwhere $f^{(j)}$ denotes the $j$-th element of $f(.)$ and $q \\in (0, 1]$.\n$\\mathcal{L}_q$ is equivalent to CCE when $q \\to 0$ and becomes MAE when $q=1$. Empirical experiments show that there exists a threshold of $q$ with which overfitting never emerges and the noisier the data the higher such a threshold should be.\nGiven true and predicted labels, $y_i, \\hat{y}_i \\in \\{0, 1\\}$ and let $u_i=y_i \\cdot \\hat{y}_i$, the\nzero-one loss\n, $\\mathcal{L}_{01}(\\mathbf{u}) = \\sum_{i=1}^n \\mathbb{1}[u_i < 0]$, is another learning subjective shown to be robust to noisy data. Minimizing the empirical risk with the zero-one loss is shown to be equivalent to minimizing the empirical adversarial (worse-case) risk (\nHu et al 2018\n). Because the worst-case risk is the upper bound of the classification risk of the clean data distribution, minimizing the worst-case risk can lead to decreased true risk, which makes the zero-one loss especially robust. However, the zero-one loss is non-differentiable and cannot be optimized directly. One solution is to approximate an\nupper bound\nof the zero-one loss and to minimize the upper bound loss instead.\nThe\nhinge loss\n, $\\mathcal{L}_\\text{hinge}(\\mathbf{u}) = \\sum_{i=1}^n \\max(0, 1 - u_i)$, defines a rough upper bound of the zero-one loss.\nLyu & Tsang (2020)\nproposed a\ncurriculum loss\n(\nCL\n), which is a tighter upper bound compared to a conventional surrogate loss like the hinge loss, $\\mathcal{L}_\\text{01}(\\mathbf{u}) \\leq \\mathcal{L}_\\text{CL}(\\mathbf{u}) \\leq \\mathcal{L}_\\text{hinge}(\\mathbf{u})$.\n$$\n\\mathcal{L}_\\text{CL}(\\mathbf{u}) = \\min_{\\mathbf{w}\\in\\{0,1\\}^n}\\max(\\sum_{i=1}^n w_i \\ell(u_i), n - \\sum_{i=1}^n w_i + \\sum_{i=1}^n\\mathbb{1}[u_i < 0])\n$$\nwhere $\\ell(u_i)$ is a base surrogate loss for the zero-one loss (e.g. hinge loss) and the optimal weighting variable $\\mathbf{w}$ is to be learned.\nGiven a label corruption rate $\\rho$, the\nnoise pruned curriculum loss\n(\nNPCL\n) is constructed based on the intuition that an ideal model should correctly classify $n(1-\\rho)$ samples with clean labels but misclassify $n\\rho$ corrupted labels. If $\\rho$ is a known prior, we would know how many samples (with largest losses) to be pruned. Assuming $\\ell(u_1) \\leq \\dots \\leq \\ell(u_n)$, then $u_{n(1-\\rho)+1} = \\dots = u_n =0$ and the following NPCL is the basic CL for only $n(1-\\rho)$ samples:\n$$\n\\text{NPCL}(\\mathbf{u}) = \\min_{\\mathbf{w}\\in\\{0,1\\}^{n(1-\\rho)}} \\max(\\sum_{i=1}^{n(1-\\rho)} w_i \\ell(u_i), n(1-\\rho) - \\sum_{i=1}^{n(1-\\rho)} w_i)\n$$\nWhen experimenting on CIFAR-10, NPCL is comparable with GCE and performs better when the noise rate increases.\nLabel Correction\n#\nSince it is known some labels are incorrect, noise-robust training can explicitly take the label correction into consideration.\nOne approach is to rely on the estimation of a noise transition matrix and use that to correct the forward or backward loss, named\nF-correction\n(\nPatrini et al. 2017\n). Let’s first assume that there are $k$ classes and the noise transition matrix $C \\in [0, 1]^{k\\times k}$ is observable and the label flipping probability does not depend on the sample input but only the label (i.e. known as random classification noise, RCN). Let $\\tilde{y}$ denote a corrupted label. Each entry of $C$ represents the probability of one label flipping to another\n1\n,\n$$\nC_{ij} = p(\\tilde{y}= j \\vert y =i, \\mathbf{x}) \\approx p(\\tilde{y}= j \\vert y =i)\n$$\nThen we can proceed a forward label correction procedure to incorporate the prior knowledge of noisy transition matrix into the prediction.\n$$\n\\begin{aligned}\n\\mathcal{L}(\\hat{p}(\\tilde{y}\\vert\\mathbf{x}), y)\n&= - \\log \\hat{p}(\\tilde{y}=i\\vert\\mathbf{x}) \\\\\n&= - \\log \\sum_{j=1}^k p(\\tilde{y}=i\\vert y=j) \\hat{p}(y=j\\vert\\mathbf{x}) \\\\\n&= - \\log \\sum_{j=1}^k C_{ji} \\hat{p}(y=j\\vert\\mathbf{x})\n\\end{aligned}\n$$\nIn matrix form, we have $\\mathcal{L}(\\hat{p}(y \\vert \\mathbf{x})) = - \\log C^\\top \\hat{p}(y \\vert \\mathbf{x})$. However, such a noise transition matrix is usually\nunknown\n. If we have access to a clean dataset, the noise matrix $C$ can be estimated (\nHendrycks et al. 2018\n) by calculating confusion matrix on the clean data. Let’s denote a clean trusted dataset as $\\mathcal{D}_c$ and a noisy dataset as $\\mathcal{D}_n$ going forward.\n$$\n\\hat{C}_{ij}\n= \\frac{1}{\\vert \\mathcal{A}_i\\vert} \\sum_{\\mathbf{x} \\in \\mathcal{A}_i} \\hat{p}(\\tilde{y}=j \\vert y=i, \\mathbf{x})\n\\approx p(\\tilde{y}=j \\vert y=i)\n$$\nwhere $\\mathcal{A}_i$ is a subset of data points from $\\mathcal{D}_c$ with label $i$.\nLet $f(x) = \\hat{p}(\\tilde{y} \\vert \\mathbf{x}; \\theta)$ and this model should be trained with $\\mathcal{L}(f(\\mathbf{x}), y)$ on clean data $\\mathcal{D}_c$ and with $\\mathcal{L}(\\hat{C}^\\top f(\\mathbf{x}), \\hat{y})$ on noisy data $\\mathcal{D}_n$.\nAlgorithm of gold loss correction (GLC), estimating the noise transition matrix with a trusted dataset. (Image source:\nHendrycks et al. 2018\n)\nIf the trusted training dataset $\\mathcal{D}_c$ gets large, we can train a neural network only on clean data and\ndistill\nits knowledge into the primary model (i.e. the final model to make predictions at test time) using corrected\npseudo labels\n(\nLi et al. 2017\n). The primary model is trained on the entire dataset, $\\mathcal{D} = \\mathcal{D}_c \\cup \\mathcal{D}_n$. Optionally the “side” information of label relations in the knowledge graph, if available, can be incorporated into distillation to help the robustness of the predictions of the network that is trained on limited data.\nThe label correction distillation works as following:\nFirst train an auxiliary model $f_c$ from the small clean dataset $\\mathcal{D}_c$ to provide a soft label for each sample $x_i$, $s_i = \\delta(f_c(\\mathbf{x}_i)/T)$ is the sigmoid activation with temperature $T$.\nBecause the clean dataset is not large, $f_c$ is likely to overfit,\nLi et al. (2017)\nturn to a knowledge graph $\\mathcal{G}$ that defines the relations in the label space and\npropagate\nthe prediction among labels accordingly. The new soft label is donated as $\\hat{s}_i = \\mathcal{G}(s_i)$.\nThe primary model $f$ is trained with predictions from $f_c$ to imitate,\n$$\n\\mathcal{L}(y_i, f(\\mathbf{x}_i)) = \\text{CE}(\\underbrace{\\lambda y_i + (1 - \\lambda) \\hat{s}_i}_\\text{pseudo label}, f(\\mathbf{x}_i))\n$$\nSample Reweighting and Selection\n#\nSome samples may be more likely to have inaccurate labels than others. Such estimation gives us intuition on which samples should be weighted less or more in the loss function. However, considering two types of biases in training data, class imbalance and noisy labels, there is actually a contradictory preference — We would prefer samples with larger loss to balance the label distribution but those with smaller loss for mitigating the potential noise. Some work (\nRen et al. 2018\n) thus argue that in order to learn general forms of training data biases, it is\nnecessary\nto have\na small unbiased validation\nto guide training. The sample reweighting methods presented in this section all assume access to a small trusted set of clean data.\nConsidering a binary classification task with random classification noise, $y, \\hat{y} \\in \\{-1, +1\\}$, the label flipping probabilities, $\\rho_{-1}, \\rho_{+1} \\in [0, 0.5)$, are defined as:\n$$\n\\rho_{-1} = P(\\tilde{y} = +1 \\vert y=-1)\\quad\\rho_{+1} = P(\\tilde{y}=-1 \\vert y =+1)\n$$\nLiu & Tao (2015)\napplies\nimportance reweighting\nto adjust the weighted distribution of observed $\\hat{y}$ to match the distribution of unobservable $y$. Let $\\mathcal{D}$ be the true data distribution and $\\mathcal{D}_\\rho$ be the corrupted version.\n$$\n\\begin{aligned}\n\\mathcal{L}_{\\ell,\\mathcal{D}}(f)\n&= \\mathbb{E}_{(\\mathbf{x},y)\\sim \\mathcal{D}}[\\ell(f(\\mathbf{x}), y)] \\\\\n&= \\mathbb{E}_{(\\mathbf{x},\\tilde{y})\\sim \\mathcal{D}_\\rho} \\Big[ \\frac{P_\\mathcal{D}(\\mathbf{x}, y=\\tilde{y})}{P_{\\mathcal{D}_\\rho}(\\mathbf{x}, \\tilde{y})} \\ell(f(\\mathbf{x}), \\tilde{y}) \\Big] \\\\\n&= \\mathbb{E}_{(\\mathbf{x},\\tilde{y})\\sim \\mathcal{D}_\\rho} \\Big[ \\frac{P_\\mathcal{D}(y=\\tilde{y} \\vert \\mathbf{x})}{P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x})} \\ell(f(\\mathbf{x}), \\tilde{y}) \\Big] & \\text{; because }P_\\mathcal{D}(\\mathbf{x})=P_{\\mathcal{D}_\\rho}(\\mathbf{x}) \\\\\n&= \\mathbb{E}_{(\\mathbf{x},\\tilde{y})\\sim \\mathcal{D}_\\rho} [ w(\\mathbf{x}, \\hat{y})\\ell(f(\\mathbf{x}), \\tilde{y}) ]\n= \\mathcal{L}_{w\\ell,\\mathcal{D}}(f)\n\\end{aligned}\n$$\nBecause,\n$$\n\\begin{aligned}\nP_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x})\n&= P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x}) P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert y=\\tilde{y}) +\nP_\\mathcal{D}(y = - \\tilde{y} \\vert \\mathbf{x}) P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert y = - \\tilde{y}) \\\\\n&= P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x}) (1 - P_{\\mathcal{D}_\\rho}(- \\tilde{y} \\vert y=\\tilde{y})) +\n(1 - P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x})) P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert y = - \\tilde{y}) \\\\\n&= P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x}) (1 - \\rho_{\\tilde{y}}) +\n(1 - P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x})) \\rho_{-\\tilde{y}} \\\\\n&= P_\\mathcal{D}(y = \\tilde{y} \\vert \\mathbf{x})(1 - \\rho_{\\tilde{y}} - \\rho_{-\\tilde{y}}) + \\rho_{-\\tilde{y}}\n\\end{aligned}\n$$\nThus the weight assigned to a noisy sample is,\n$$\nw(x, \\tilde{y})\n= \\frac{P_\\mathcal{D}(y=\\tilde{y} \\vert \\mathbf{x})}{P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x})}\n= \\frac{P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x}) - \\rho_{-\\tilde{y}}}{(1-\\rho_0-\\rho_1) P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x})}\n$$\nwhere $P_{\\mathcal{D}_\\rho}(\\tilde{y} \\vert \\mathbf{x})$ can be estimated using a simple logistic regression, but estimating the note rates is more challenging. Naive cross-validation can work out but is costly as the quality depends on the amount of trusted labels available. The paper approximates the upper bounds for noise rates first, $\\rho_\\tilde{y} \\leq P_{\\mathcal{D}_\\rho}(- \\tilde{y} \\vert \\mathbf{x})$ and then use a mild assumption to efficiently estimate them, $\\hat{\\rho}_{\\tilde{y}} = \\min_{\\mathbf{x} \\in {\\mathbf{x}_1, \\dots, \\mathbf{x}_n}} \\hat{P}_{\\mathcal{D}_\\rho}(- \\tilde{y} \\vert \\mathbf{x})$. In their experiments, the advantage of importance reweighting only varies across datasets and is more beneficial when the noise rates are high in general.\nSample reweighting schemes can be learned by a separate network.\nLearning to reweight\n(\nL2R\n;\nRen et al. 2018\n) is a meta-learning approach to directly optimize the weights in pursuit of best validation performance on a known set of clean data. Each example gets assigned with the weight based on its gradient direction. The weighted loss to minimize $\\theta^*(\\mathbf{w})$ involves a set of training weights $\\{w_i\\}_{i=1}^n$ as unknown hyperparameters. These sample training weights $w_i$ are learned to minimize the loss on this unbiased validate set, $\\mathcal{D}_c = \\{x^\\text{valid}_j\\}_{j=1}^m$.\n$$\n\\begin{aligned}\n\\theta^{*}(\\mathbf{w}) &= \\arg\\min_\\theta \\sum_{i=1}^n w_i f(x_i; \\theta) \\\\\n\\text{where optimal }\\mathbf{w}^{*} &= \\arg\\min_{\\mathbf{w}, \\mathbf{w} \\geq \\mathbf{0}} \\frac{1}{m} \\sum_{j=1}^m f(\\mathbf{x}^\\text{valid}_j; \\theta^{*}(\\mathbf{w}))\n\\end{aligned}\n$$\nThe learning process involves two nested loops of optimization, so pretty expensive, 3x training time.\nIllustration of updates implemented by second order\nautomatic differentiation\n. (Image source:\nRen et al. 2018\n)\nThey ran experiments on (1) two-class MNIST to test the robustness of L2R when the class distribution is imbalanced and (2) CIFAR-10 with noisy labels. L2R is shown to be better than other baseline methods at the time on both tasks.\nLeft: Imbalanced classes on MNIST (class 4 and 9); Right: Effect of the number of clean samples. Task is on CIFAR-10 with 40% of data flipped to label 3. (Image source:\nRen et al. 2018\n)\nMentorNet\n(\nJiang et al. 2018\n) uses teach-student curriculum learning to weight data. It incorporates two different networks, a mentor and a student. The mentor network provides a data-driven curriculum (i.e. sample training weighting scheme) for the student to focus on learning likely correct labels.\nLet $g_\\psi$ be the MentorNet parameterized by $\\psi$ , $f_\\theta$ be the StudentNet parametrized by $\\theta$ and $G$ be a predefined curriculum parameterized by $\\lambda$. Given the training data $\\mathcal{D} = \\{(\\mathbf{x}_i, y_i)\\}_{i=1}^n$ for a $k$-class classification task, the MentorNet needs to predict a time-varying latent weight variable $\\mathbf{w} \\in [0, 1]^{n \\times k}$ to guide the learning of StudentNet, taking an intermediate feature processed by StudentNet $f$ , $\\mathbf{z}_i = \\phi_{f_\\theta}(\\mathbf{x}_i, y_i)$:\n$$\ng_{\\psi^{*}}(\\mathbf{z}_i) = \\arg\\min_{w_i \\in [0,1]} \\mathcal{L}(\\theta, \\mathbf{w}), \\forall i \\in [1, n]\n$$\nStudentNet learns to minimize the following learning objective,\n$$\n\\begin{aligned}\n\\mathcal{L}(\\theta, \\mathbf{w})\n&= \\frac{1}{n}\\sum_{i=1}^n \\mathbf{w}_i^\\top \\ell(y_i, f_\\theta(\\mathbf{x}_i)) + G_\\lambda(\\mathbf{w}) + \\alpha |\\theta|^2_2 \\\\\n&= \\frac{1}{n}\\sum_{i=1}^n g_\\psi(\\mathbf{z}_i)^\\top \\ell_i + G_\\lambda(\\mathbf{w}) + \\alpha |\\theta|^2_2 & \\text{; Let }\\ell_i = \\ell(y_i, f_\\theta(\\mathbf{x}_i)) \\\\\n\\end{aligned}\n$$\nThe mentor network $g_\\psi$ is trained with cross entropy on the input $(\\phi_{f_\\theta}(\\mathbf{x}_i, y_i), w^{*}_i)$ , where $v^*_i=1$ if $y_i$ is known to be a correct label, otherwise 0. The architecture of MentorNet does not have to be very complicated. In the paper, they adopted a LSTM layer to capture the prediction variance in time.\nModel architecture of MentorNet and StudentNet which are trained simultaneously, where MentorNet predicts the sample weights for StudentNet to train on. (Image source:\nJiang et al. 2018\n)\nDifferent from MentorNet where one network explicitly learns weighting scheme and curriculum for the other network,\nCo-teaching\n(\nHan et al. 2018\n) trains two neural networks, $f_1$ and $f_2$, simultaneously and lets them teach each other by feeding data to each other selectively. Co-teaching consists of three steps:\nFirst, each network feeds forward the current mini-batch and selects samples with potentially clean labels;\nThen two networks exchange information on which samples in the batch should be used for training. Small-loss instances are selected as they are more likely to be associated with correct labels. The percentage of the batch to select is determined by a time-dependent function $R(T)$. The value of $R(T)$ decreases in time because the network is more likely to overfit and memorize noisy labels as training progresses and thus we use a smaller sampling percentage to keep the selected data quality high.\nFinally, each network runs back-propagation updates with the data selected by its peer.\nAccording to their experiments, co-teaching performs better than\nF-correction\nwhere the noise rates are high or the corruption transition matrix is not symmetric.\nAlgorithm of co-teaching in which two networks are trained separately in parallel and each selects samples for the other to train on. (Image source:\nHan et al. 2018\n)\nCitation\n#\nCited as:\nWeng, Lilian. (Apr 2022). Learning with not enough data part 3: data generation. Lil’Log. https://lilianweng.github.io/posts/2022-04-15-data-gen/.\nOr\n@article{weng2022datagen,\n title = \"Learning with not Enough Data Part 3: Data Generation\",\n author = \"Weng, Lilian\",\n journal = \"Lil'Log\",\n year = \"2022\",\n month = \"Apr\",\n url = \"https://lilianweng.github.io/posts/2022-04-15-data-gen/\"\n}\nReference\n#\n[1] Zhang et al.\n“Adversarial AutoAgument”\nICLR 2020.\n[2] Kumar et al.\n“Data Augmentation using Pre-trained Transformer Models.”\nAACL 2020 Workshop.\n[3] Anaby-Tavor et al.\n“Not enough data? Deep learning to rescue!”\nAAAI 2020.\n[4] Wang et al.\n“Want To Reduce Labeling Cost? GPT-3 Can Help.”\nEMNLP 2021.\n[5] Wang et al.\n“Towards Zero-Label Language Learning.”\narXiv preprint arXiv:2109.09193 (2021).\n[6] Schick & Schutze.\nGenerating Datasets with Pretrained Language Models.\"\nEMNLP 2021.\n[7] Han et al.\n“Unsupervised Neural Machine Translation with Generative Language Models Only.”\narXiv preprint arXiv:2110.05448 (2021).\n[8] Guo et al.\n“Augmenting data with mixup for sentence classification: An empirical study.”\narXiv preprint arXiv:1905.08941 (2019).\n[9] Ekin D. Cubuk et al.\n“AutoAugment: Learning augmentation policies from data.”\narXiv preprint arXiv:1805.09501 (2018).\n[10] Daniel Ho et al.\n“Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules.”\nICML 2019.\n[11] Cubuk & Zoph et al.\n“RandAugment: Practical automated data augmentation with a reduced search space.”\narXiv preprint arXiv:1909.13719 (2019).\n[12] Zhang et al.\n“mixup: Beyond Empirical Risk Minimization.”\nICLR 2017.\n[13] Yun et al.\n“CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features.”\nICCV 2019.\n[14] Kalantidis et al.\n“Mixing of Contrastive Hard Negatives”\nNeuriPS 2020.\n[15] Wei & Zou.\n“EDA: Easy data augmentation techniques for boosting performance on text classification tasks.”\nEMNLP-IJCNLP 2019.\n[16] Kobayashi.\n“Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations.”\nNAACL 2018\n[17] Fang et al.\n“CERT: Contrastive self-supervised learning for language understanding.”\narXiv preprint arXiv:2005.12766 (2020).\n[18] Gao et al.\n“SimCSE: Simple Contrastive Learning of Sentence Embeddings.”\narXiv preprint arXiv:2104.08821 (2020). [\ncode\n]\n[19] Shen et al.\n“A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation.”\narXiv preprint arXiv:2009.13818 (2020) [\ncode\n]\n[20] Wang & van den Oord.\n“Multi-Format Contrastive Learning of Audio Representations.”\nNeuriPS Workshop 2020.\n[21] Wu et al.\n“Conditional BERT Contextual Augmentation”\narXiv preprint arXiv:1812.06705 (2018).\n[22 Zhu et al.\n“FreeLB: Enhanced Adversarial Training for Natural Language Understanding.”\nICLR 2020.\n[23] Affinity and Diversity: Quantifying Mechanisms of Data Augmentation\nGontijo-Lopes et al. 2020 (\nhttps://arxiv.org/abs/2002.08973\n)\n[24] Song et al.\n“Learning from Noisy Labels with Deep Neural Networks: A Survey.”\nTNNLS 2020.\n[25] Zhang & Sabuncu.\n“Generalized cross entropy loss for training deep neural networks with noisy labels.”\nNeuriPS 2018.\n[26] Goldberger & Ben-Reuven.\n“Training deep neural-networks using a noise adaptation layer.”\nICLR 2017.\n[27] Sukhbaatar et al.\n“Training convolutional networks with noisy labels.”\nICLR Workshop 2015.\n[28] Patrini et al.\n“Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach”\nCVPR 2017.\n[29] Hendrycks et al.\n“Using trusted data to train deep networks on labels corrupted by severe noise.”\nNeuriPS 2018.\n[30] Zhang & Sabuncu.\n“Generalized cross entropy loss for training deep neural networks with noisy labels.”\nNeuriPS 2018.\n[31] Lyu & Tsang.\n“Curriculum loss: Robust learning and generalization against label corruption.”\nICLR 2020.\n[32] Han et al.\n“Co-teaching: Robust training of deep neural networks with extremely noisy labels.”\nNeuriPS 2018. (\ncode\n)\n[33] Ren et al.\n“Learning to reweight examples for robust deep learning.”\nICML 2018.\n[34] Jiang et al.\n“MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels.”\nICML 2018.\n[35] Li et al.\n“Learning from noisy labels with distillation.”\nICCV 2017.\n[36] Liu & Tao.\n“Classification with noisy labels by importance reweighting.”\nTPAMI 2015.\n[37] Ghosh, et al.\n“Robust loss functions under label noise for deep neural networks.”\nAAAI 2017.\n[38] Hu et al.\n“Does Distributionally Robust Supervised Learning Give Robust Classifiers? “\nICML 2018.\n$y=i$ is not a technically correct way to annotate a label being a certain value, since we usually use one-hot encoding (i.e. $\\mathbf{y} = \\mathbf{e}_i$). We use this form for simplicity.\n↩︎", "metadata": { "title": "Learning with not Enough Data Part 3: Data Generation | Lil'Log", "description": "Here comes the Part 3 on learning with not enough data (Previous: Part 1 and Part 2). Let’s consider two approaches for generating synthetic data for training.\n\nAugmented data. Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a previous post on contrastive learning. For the sake of post completeness, I duplicate the section on data augmentation here with some edits.\nNew data. Given few or even no data points, we can rely on powerful pretrained models to generate a number of new data points. This is especially true in recent years given the fast progress in large pretrained language models (LM). Few shot prompting is shown to be effective for LM to learn within context without extra training.\n\nData Augmentation\nThe goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.", "keywords": "data, data-generation, data-augmentation, data-quality", "author": "Lilian Weng", "og:title": "Learning with not Enough Data Part 3: Data Generation", "og:description": "Here comes the Part 3 on learning with not enough data (Previous: Part 1 and Part 2). Let’s consider two approaches for generating synthetic data for training.\n\nAugmented data. Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a previous post on contrastive learning. For the sake of post completeness, I duplicate the section on data augmentation here with some edits.\nNew data. Given few or even no data points, we can rely on powerful pretrained models to generate a number of new data points. This is especially true in recent years given the fast progress in large pretrained language models (LM). Few shot prompting is shown to be effective for LM to learn within context without extra training.\n\nData Augmentation\nThe goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.", "og:type": "article", "og:url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/", "twitter:card": "summary", "twitter:title": "Learning with not Enough Data Part 3: Data Generation", "twitter:description": "Here comes the Part 3 on learning with not enough data (Previous: Part 1 and Part 2). Let’s consider two approaches for generating synthetic data for training.\n\nAugmented data. Given a set of existing training samples, we can apply a variety of augmentation, distortion and transformation to derive new data points without losing the key attributes. We have covered a bunch of augmentation methods on text and images in a previous post on contrastive learning. For the sake of post completeness, I duplicate the section on data augmentation here with some edits.\nNew data. Given few or even no data points, we can rely on powerful pretrained models to generate a number of new data points. This is especially true in recent years given the fast progress in large pretrained language models (LM). Few shot prompting is shown to be effective for LM to learn within context without extra training.\n\nData Augmentation\nThe goal of data augmentation is to modify the input format (e.g. text wording, visual appearance) while the semantic meaning stays unchanged.", "article:section": "posts", "article:published_time": "2022-04-15T15:10:30-07:00", "article:modified_time": "2022-04-15T15:10:30-07:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/1cc4df7eb2c7/code/body-000.txt", "chars": 257, "language": "text" } ], "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/EDA-exp1.png", "bytes": 299081, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-b1e26be047c763592a3a.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/EDA-exp1.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GPT3-data-gen.png", "bytes": 178022, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-364dc68a08807b5baf8a.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GPT3-data-gen.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GPT3-data-gen-exp.png", "bytes": 979053, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-2a8613e8ff2f07c9bbde.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GPT3-data-gen-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/LAMBADA.png", "bytes": 163034, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-1e49c821edae4f24f95b.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/LAMBADA.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/UDG.png", "bytes": 396874, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-773377f93dad7a113410.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/UDG.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/UDG-exp.png", "bytes": 103983, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-82e84409bcacc2fc600e.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/UDG-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/back-translation-data-gen.png", "bytes": 161004, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-3800c7c1476957d08f84.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/back-translation-data-gen.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/back-translation-data-gen-exp.png", "bytes": 107466, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-9ca541330b72888db6f9.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/back-translation-data-gen-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/affinity-diversity.png", "bytes": 257673, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-6945c914870c32780b47.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/affinity-diversity.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/noise-adaptation-layer.png", "bytes": 193590, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-1e20893406355a07939d.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/noise-adaptation-layer.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GLC.png", "bytes": 283685, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-6185b7cd92536235d8de.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/GLC.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/L2R-backprop.png", "bytes": 241069, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-67e6e4a22214ef58e33c.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/L2R-backprop.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/L2R-exp.png", "bytes": 812915, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-7d52e35a3da415faf3df.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/L2R-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/MentorNet.png", "bytes": 260104, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-e2db6011ba9c735f3718.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/MentorNet.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/co-teaching.png", "bytes": 313297, "reused": false, "local_path": "attachments/1cc4df7eb2c7/images/body-cb3814a9fb4722b3289a.png", "src": "https://lilianweng.github.io/posts/2022-04-15-data-gen/co-teaching.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/", "label": "Part 1", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2022-02-20-active-learning/", "label": "Part 2", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2021-05-31-contrastive/", "label": "previous post", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2019-01-31-lm/", "label": "language models (LM)", "type": "a" }, { "href": "https://pillow.readthedocs.io/en/stable/reference/ImageOps.html", "label": "PIL.ImageOps", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2019-05-05-domain-randomization/", "label": "AutoAugment", "type": "a" }, { "href": "https://arxiv.org/abs/1805.09501", "label": "Cubuk, et al. 2018", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2020-08-06-nas/", "label": "neural architecture search", "type": "a" }, { "href": "https://arxiv.org/abs/1912.11188", "label": "Zhang, et al 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1909.13719", "label": "Cubuk et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1905.05393", "label": "Ho et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1711.09846", "label": "Jaderberg et al, 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1904.12848", "label": "Xie et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1710.09412", "label": "Zhang et al., 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1905.04899", "label": "Yun et al., 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1708.04552", "label": "DeVries & Taylor 2017", "type": "a" }, { "href": "https://arxiv.org/abs/2010.01028", "label": "Kalantidis et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1901.11196", "label": "Wei & Zou 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1805.06201", "label": "Kobayashi, 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1812.06705", "label": "Xing Wu et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1905.08941", "label": "Guo et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1909.11764", "label": "Zhu et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/2103.06508", "label": "Wang & van den Oord (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/2104.08821", "label": "Guo et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2009.13818", "label": "Shen et al. (2020)", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2017-08-20-gan/", "label": "GAN", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2018-08-12-vae/", "label": "VAE", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2018-10-13-flow-models/", "label": "flow", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/", "label": "diffusion", "type": "a" }, { "href": "https://arxiv.org/abs/2108.13487", "label": "Wang et al. (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/1911.03118", "label": "Anaby-Tavor et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/2003.02245", "label": "Kumar et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2109.09193", "label": "Wang et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2104.07540", "label": "Schick & Schutze (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/2110.05448", "label": "Han et al (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/2002.08973", "label": "Gontijo-Lopes et al. (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/2007.08199", "label": "survey paper (Song et al. 2021)", "type": "a" }, { "href": "https://arxiv.org/abs/1406.2080", "label": "Sukhbaatar et al. 2015", "type": "a" }, { "href": "https://openreview.net/forum?id=H12GRgcxg", "label": "Goldberger & Ben-Reuven, 2017", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm", "label": "EM algorithm", "type": "a" }, { "href": "https://arxiv.org/abs/1712.09482", "label": "Ghosh et al. 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1805.07836", "label": "Zhang & Sabuncu (2018)", "type": "a" }, { "href": "https://arxiv.org/abs/1611.02041", "label": "Hu et al 2018", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Hinge_loss", "label": "hinge loss", "type": "a" }, { "href": "https://arxiv.org/abs/1905.10045", "label": "Lyu & Tsang (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/1609.03683", "label": "Patrini et al. 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1802.05300", "label": "Hendrycks et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1703.02391", "label": "Li et al. 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1803.09050", "label": "Ren et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1411.7718", "label": "Liu & Tao (2015)", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Automatic_differentiation", "label": "automatic differentiation", "type": "a" }, { "href": "https://arxiv.org/abs/1712.05055", "label": "Jiang et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/1804.06872", "label": "Han et al. 2018", "type": "a" }, { "href": "https://arxiv.org/abs/2005.12766", "label": "“CERT: Contrastive self-supervised learning for language understanding.”", "type": "a" }, { "href": "https://github.com/princeton-nlp/SimCSE", "label": "code", "type": "a" }, { "href": "https://github.com/dinghanshen/cutoff", "label": "code", "type": "a" }, { "href": "https://github.com/bhanML/Co-teaching", "label": "code", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:51:25Z", "source_url": "https://lilianweng.github.io/posts/2022-04-15-data-gen/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/1805.09501", "bytes": 1138440, "reused": false, "local_path": "papers/repair/9a0542ec64439d672cc9.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1912.11188", "bytes": 759445, "reused": false, "local_path": "papers/repair/4f4d45c6d87c67d3af9a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.13719", "bytes": 1263406, "reused": true, "local_path": "papers/repair/c267ff367c4770ddb6c0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.05393", "bytes": 873126, "reused": false, "local_path": "papers/repair/4634b6280b855d570ac8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1711.09846", "bytes": 2370925, "reused": false, "local_path": "papers/repair/3bbdbf9c8931b9ea9ea0.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1904.12848", "bytes": 1625724, "reused": true, "local_path": "papers/repair/17af125202b45606d9ad.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1710.09412", "bytes": 740562, "reused": true, "local_path": "papers/repair/7af2d52fbf38a69ec251.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.04899", "bytes": 1682836, "reused": false, "local_path": "papers/repair/fec62c4fe8efdeebd877.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1708.04552", "bytes": 741919, "reused": false, "local_path": "papers/repair/c3d681ed19bf88f43a98.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2010.01028", "bytes": 970998, "reused": false, "local_path": "papers/repair/d4d3c2412459021d347a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1901.11196", "bytes": 365049, "reused": true, "local_path": "papers/repair/389e34d78a0045e6d861.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1805.06201", "bytes": 722323, "reused": false, "local_path": "papers/repair/aebd3f77254c72e012a3.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1812.06705", "bytes": 431373, "reused": false, "local_path": "papers/repair/3e737149c9ea8568055b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.08941", "bytes": 345640, "reused": false, "local_path": "papers/repair/646a76cd251b8253359a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.11764", "bytes": 257462, "reused": false, "local_path": "papers/repair/f5ea669209a03760ab22.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2103.06508", "bytes": 601155, "reused": false, "local_path": "papers/repair/a3681dc71a79ffecd499.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2104.08821", "bytes": 1672580, "reused": false, "local_path": "papers/repair/e7a40652bdd04cd4b25f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2009.13818", "bytes": 398629, "reused": false, "local_path": "papers/repair/40a2d9b786e78e3cf30e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2108.13487", "bytes": 5884494, "reused": false, "local_path": "papers/repair/13a1e2b1cf2e73968c6b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1911.03118", "bytes": 372575, "reused": false, "local_path": "papers/repair/e8abfe05a235151d6c3e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2003.02245", "bytes": 250909, "reused": false, "local_path": "papers/repair/ba7634f079f15d2367dd.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2109.09193", "bytes": 359282, "reused": false, "local_path": "papers/repair/5b433ceacb396f0baf99.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2104.07540", "bytes": 402533, "reused": false, "local_path": "papers/repair/ffa07cbf93071d00f1ed.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2110.05448", "bytes": 386252, "reused": false, "local_path": "papers/repair/62c1d70878370a3a749f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.08973", "bytes": 1161444, "reused": false, "local_path": "papers/repair/cfe2f6f56fb90471ae4d.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2007.08199", "bytes": 1299522, "reused": false, "local_path": "papers/repair/e31349c322026bf09dbb.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1406.2080", "bytes": 1181382, "reused": false, "local_path": "papers/repair/bd04e7310e34c7124557.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1712.09482", "bytes": 1146169, "reused": false, "local_path": "papers/repair/c7b83a3603919eb61881.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1805.07836", "bytes": 569770, "reused": false, "local_path": "papers/repair/1a42ffbbf4e704947ae3.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1611.02041", "bytes": 687019, "reused": false, "local_path": "papers/repair/ae376c87ceb11d4ea40f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.10045", "bytes": 2142637, "reused": false, "local_path": "papers/repair/75e3f3f69d9794770568.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1609.03683", "bytes": 763555, "reused": false, "local_path": "papers/repair/e8309ba701b2977efaca.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1802.05300", "bytes": 1049334, "reused": false, "local_path": "papers/repair/9237191e681f5b67f6cc.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1703.02391", "bytes": 446285, "reused": false, "local_path": "papers/repair/e7f30c1dcfc80c923753.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1803.09050", "bytes": 3044218, "reused": false, "local_path": "papers/repair/40a7062040ed9fa7132b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1411.7718", "bytes": 634569, "reused": false, "local_path": "papers/repair/d8403b69812a8c4b8935.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1712.05055", "bytes": 827072, "reused": false, "local_path": "papers/repair/9788972c8ad31eb5bbb5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1804.06872", "bytes": 1984230, "reused": false, "local_path": "papers/repair/ce399494158e733915af.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2005.12766", "bytes": 412104, "reused": true, "local_path": "papers/repair/bae8e6b74f13390eb0a6.pdf" } ], "actual_video_files": [], "body_chars": 40655, "expected_body_images": 15, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/1cc4df7eb2c7/pages/source.html", "local_html": "attachments/1cc4df7eb2c7/pages/article.html", "body_base_path": "attachments/1cc4df7eb2c7/pages/", "body_markdown_path": "attachments/1cc4df7eb2c7/pages/article.md" }