{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "Learning with not Enough Data Part 1: Semi-Supervised Learning",
  "keywords": "半监督学习,少数据,标注",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/",
  "publish_date": "2021-12",
  "corpus_stage": "数据处理",
  "notes": "半监督学习系列综述之一",
  "id": "76583c4985bd",
  "collected_at": "2026-10-02T14:13:01",
  "title": "Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Log",
  "body_markdown": "When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.\nPre-training + fine-tuning\n: Pre-train a powerful task-agnostic model on a large unsupervised data corpus, e.g.\npre-training LMs\non free text, or pre-training vision models on unlabelled images via\nself-supervised learning\n, and then fine-tune it on the downstream task with a small set of labeled samples.\nSemi-supervised learning\n: Learn from the labelled and unlabeled samples together. A lot of research has happened on vision tasks within this approach.\nActive learning\n: Labeling is expensive, but we still want to collect more given a cost budget. Active learning learns to select most valuable unlabeled samples to be collected next and helps us act smartly with a limited budget.\nPre-training + dataset auto-generation\n: Given a capable pre-trained model, we can utilize it to auto-generate a lot more labeled samples. This has been especially popular within the language domain driven by the success of few-shot learning.\nI plan to write a series of posts on the topic of “Learning with not enough data”. Part 1 is on\nSemi-Supervised Learning\n.\nWhat is semi-supervised learning?\n#\nSemi-supervised learning uses both labeled and unlabeled data to train a model.\nInterestingly most existing literature on semi-supervised learning focuses on vision tasks. And instead pre-training + fine-tuning is a more common paradigm for language tasks.\nAll the methods introduced in this post have a loss combining two parts: $\\mathcal{L} = \\mathcal{L}_s +  \\mu(t) \\mathcal{L}_u$. The supervised loss $\\mathcal{L}_s$ is easy to get given all the labeled examples. We will focus on how the unsupervised loss $\\mathcal{L}_u$ is designed. A common choice of the weighting term $\\mu(t)$ is a ramp function increasing the importance of $\\mathcal{L}_u$ in time, where $t$ is the training step.\nDisclaimer\n: The post is not gonna cover semi-supervised methods with focus on model architecture modification. Check\nthis survey\nfor how to use generative models and graph-based methods in semi-supervised learning.\nNotations\n#\nSymbol\nMeaning\n$L$\nNumber of unique labels.\n$(\\mathbf{x}^l, y) \\sim \\mathcal{X}, y \\in \\{0, 1\\}^L$\nLabeled dataset. $y$ is a one-hot representation of the true label.\n$\\mathbf{u} \\sim \\mathcal{U}$\nUnlabeled dataset.\n$\\mathcal{D} = \\mathcal{X} \\cup \\mathcal{U}$\nThe entire dataset, including both labeled and unlabeled examples.\n$\\mathbf{x}$\nAny sample which can be either labeled or unlabeled.\n$\\bar{\\mathbf{x}}$\n$\\mathbf{x}$ with augmentation applied.\n$\\mathbf{x}_i$\nThe $i$-th sample.\n$\\mathcal{L}$, $\\mathcal{L}_s$, $\\mathcal{L}_u$\nLoss, supervised loss, and unsupervised loss.\n$\\mu(t)$\nThe unsupervised loss weight, increasing in time.\n$p(y \\vert \\mathbf{x}), p_\\theta(y \\vert \\mathbf{x})$\nThe conditional probability over the label set given the input.\n$f_\\theta(.)$\nThe implemented neural network with weights $\\theta$, the model that we want to train.\n$\\mathbf{z} = f_\\theta(\\mathbf{x})$\nA vector of logits output by $f$.\n$\\hat{y} = \\text{softmax}(\\mathbf{z})$\nThe predicted label distribution.\n$D[.,.]$\nA distance function between two distributions, such as MSE, cross entropy, KL divergence, etc.\n$\\beta$\nEMA weighting hyperparameter for\nteacher\nmodel weights.\n$\\alpha, \\lambda$\nParameters for MixUp, $\\lambda \\sim \\text{Beta}(\\alpha, \\alpha)$.\n$T$\nTemperature for sharpening the predicted distribution.\n$\\tau$\nA confidence threshold for selecting the qualified prediction.\nHypotheses\n#\nSeveral hypotheses have been discussed in literature to support certain design decisions in semi-supervised learning methods.\nH1:\nSmoothness Assumptions\n: If two data samples are close in a high-density region of the feature space, their labels should be the same or very similar.\nH2:\nCluster Assumptions\n: The feature space has both dense regions and sparse regions. Densely grouped data points naturally form a cluster. Samples in the same cluster are expected to have the same label. This is a small extension of H1.\nH3:\nLow-density Separation Assumptions\n: The decision boundary between classes tends to be located in the sparse, low density regions, because otherwise the decision boundary would cut a high-density cluster into two classes, corresponding to two clusters, which invalidates H1 and H2.\nH4:\nManifold Assumptions\n: The high-dimensional data tends to locate on a low-dimensional manifold. Even though real-world data might be observed in very high dimensions (e.g. such as images of real-world objects/scenes), they actually can be captured by a lower dimensional manifold where certain attributes are captured and similar points are grouped closely (e.g. images of real-world objects/scenes are not drawn from a uniform distribution over all pixel combinations). This enables us to learn a more efficient representation for us to discover and measure similarity between unlabeled data points. This is also the foundation for representation learning. [see\na helpful link\n].\nConsistency Regularization\n#\nConsistency Regularization\n, also known as\nConsistency Training\n, assumes that randomness within the neural network (e.g. with Dropout) or data augmentation transformations should not modify model predictions given the same input. Every method in this section has a consistency regularization loss as $\\mathcal{L}_u$.\nThis idea has been adopted in several\nself-supervised\nlearning\nmethods, such as SimCLR, BYOL, SimCSE, etc. Different augmented versions of the same sample should result in the same representation.\nCross-view training\nin language modeling and multi-view learning in self-supervised learning all share the same motivation.\nΠ-model\n#\nOverview of the Π-model. Two versions of the same input with different stochastic augmentation and dropout masks pass through the network and the outputs are expected to be consistent. (Image source:\nLaine & Aila (2017)\n)\nSajjadi et al. (2016)\nproposed an unsupervised learning loss to minimize the difference between two passes through the network with stochastic transformations (e.g. dropout, random max-pooling) for the same data point. The label is not explicitly used, so the loss can be applied to unlabeled dataset.\nLaine & Aila (2017)\nlater coined the name,\nΠ-Model\n, for such a setup.\n$$\n\\mathcal{L}_u^\\Pi = \\sum_{\\mathbf{x} \\in \\mathcal{D}} \\text{MSE}(f_\\theta(\\mathbf{x}), f'_\\theta(\\mathbf{x}))\n$$\nwhere $f’$ is the same neural network with different stochastic augmentation or dropout masks applied. This loss utilizes the entire dataset.\nTemporal ensembling\n#\nOverview of Temporal Ensembling. The per-sample EMA label prediction is the learning target. (Image source:\nLaine & Aila (2017)\n)\nΠ-model requests the network to run two passes per sample, doubling the computation cost. To reduce the cost,\nTemporal Ensembling\n(\nLaine & Aila 2017\n) maintains an exponential moving average (EMA) of the model prediction in time per training sample $\\tilde{\\mathbf{z}}_i$ as the learning target, which is only evaluated and updated once per epoch. Because the ensemble output $\\tilde{\\mathbf{z}}_i$ is initialized to $\\mathbf{0}$, it is normalized by $(1-\\alpha^t)$ to correct this startup bias. Adam optimizer has such\nbias correction\nterms for the same reason.\n$$\n\\tilde{\\mathbf{z}}^{(t)}_i = \\frac{\\alpha \\tilde{\\mathbf{z}}^{(t-1)}_i + (1-\\alpha) \\mathbf{z}_i}{1-\\alpha^t}\n$$\nwhere $\\tilde{\\mathbf{z}}^{(t)}$ is the ensemble prediction at epoch $t$ and $\\mathbf{z}_i$ is the model prediction in the current round. Note that since $\\tilde{\\mathbf{z}}^{(0)} = \\mathbf{0}$, with correction, $\\tilde{\\mathbf{z}}^{(1)}$ is simply equivalent to $\\mathbf{z}_i$ at epoch 1.\nMean teachers\n#\nOverview of the Mean Teacher framework. (Image source:\nTarvaninen & Valpola, 2017\n)\nTemporal Ensembling keeps track of an EMA of label predictions for each training sample as a learning target. However, this label prediction only changes\nevery epoch\n, making the approach clumsy when the training dataset is large.\nMean Teacher\n(\nTarvaninen & Valpola, 2017\n) is proposed to overcome the slowness of target update by tracking the moving average of model weights instead of model outputs. Let’s call the original model with weights $\\theta$ as the\nstudent\nmodel and the model with moving averaged weights $\\theta’$ across consecutive student models as the\nmean teacher\n: $\\theta’ \\gets \\beta \\theta’ + (1-\\beta)\\theta$\nThe consistency regularization loss is the distance between predictions by the student and teacher and the student-teacher gap should be minimized. The mean teacher is expected to provide more accurate predictions than the student. It got confirmed in the empirical experiments, as shown in\nClassification error on SVHN of Mean Teacher and the Π Model. The mean teacher (in orange) has better performance than the student model (in blue). (Image source:\nTarvaninen & Valpola, 2017\n)\nAccording to their ablation studies,\nInput augmentation (e.g. random flips of input images, Gaussian noise) or student model dropout is necessary for good performance. Dropout is not needed on the teacher model.\nThe performance is sensitive to the EMA decay hyperparameter $\\beta$. A good strategy is to use a small $\\beta=0.99$ during the ramp up stage and a larger $\\beta=0.999$ in the later stage when the student model improvement slows down.\nThey found that MSE as the consistency cost function performs better than other cost functions like KL divergence.\nNoisy samples as learning targets\n#\nSeveral recent consistency training methods learn to minimize prediction difference between the original unlabeled sample and its corresponding augmented version. It is quite similar to the Π-model but the consistency regularization loss is\nonly\napplied to the unlabeled data.\nConsistency training with noisy samples.\nAdversarial Training (\nGoodfellow et al. 2014\n) applies adversarial noise onto the input and trains the model to be robust to such adversarial attack. The setup works in supervised learning,\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{adv}(\\mathbf{x}^l, \\theta) &= D[q(y\\mid \\mathbf{x}^l), p_\\theta(y\\mid \\mathbf{x}^l + r_\\text{adv})] \\\\\nr_\\text{adv} &= {\\arg\\max}_{r; \\|r\\| \\leq \\epsilon} D[q(y\\mid \\mathbf{x}^l), p_\\theta(y\\mid \\mathbf{x}^l + r_\\text{adv})] \\\\\nr_\\text{adv} &\\approx \\epsilon \\frac{g}{\\|g\\|_2} \\approx \\epsilon\\text{sign}(g)\\quad\\text{where }g = \\nabla_{r} D[y, p_\\theta(y\\mid \\mathbf{x}^l + r)]\n\\end{aligned}\n$$\nwhere $q(y \\mid \\mathbf{x}^l)$ is the true distribution, approximated by one-hot encoding of the ground truth label, $y$. $p_\\theta(y \\mid \\mathbf{x}^l)$ is the model prediction. $D[.,.]$ is a distance function measuring the divergence between two distributions.\nVirtual Adversarial Training\n(\nVAT\n;\nMiyato et al. 2018\n) extends the idea to work in semi-supervised learning. Because $q(y \\mid \\mathbf{x}^l)$ is unknown, VAT replaces it with the current model prediction for the original input with the current weights $\\hat{\\theta}$.  Note that $\\hat{\\theta}$ is a fixed copy of model weights, so there is no gradient update on $\\hat{\\theta}$.\n$$\n\\begin{aligned}\n\\mathcal{L}_u^\\text{VAT}(\\mathbf{x}, \\theta) &= D[p_{\\hat{\\theta}}(y\\mid \\mathbf{x}), p_\\theta(y\\mid \\mathbf{x} + r_\\text{vadv})] \\\\\nr_\\text{vadv} &= {\\arg\\max}_{r; \\|r\\| \\leq \\epsilon} D[p_{\\hat{\\theta}}(y\\mid \\mathbf{x}), p_\\theta(y\\mid \\mathbf{x} + r)]\n\\end{aligned}\n$$\nThe VAT loss applies to both labeled and unlabeled samples. It is a negative smoothness measure of the current model’s prediction manifold at each data point. The optimization of such loss motivates the manifold to be smoother.\nInterpolation Consistency Training\n(\nICT\n;\nVerma et al. 2019\n) enhances the dataset by adding more interpolations of data points and expects the model prediction to be consistent with interpolations of the corresponding labels. MixUp (\nZheng et al. 2018\n) operation mixes two images via a simple weighted sum and combines it with label smoothing. Following the idea of MixUp, ICT expects the prediction model to produce a label on a mixup sample to match the interpolation of predictions of corresponding inputs:\n$$\n\\begin{aligned}\n\\text{mixup}_\\lambda (\\mathbf{x}_i, \\mathbf{x}_j) &= \\lambda \\mathbf{x}_i + (1-\\lambda)\\mathbf{x}_j \\\\\np(\\text{mixup}_\\lambda (y \\mid \\mathbf{x}_i, \\mathbf{x}_j)) &\\approx \\lambda p(y \\mid \\mathbf{x}_i) + (1-\\lambda) p(y \\mid \\mathbf{x}_j)\n\\end{aligned}\n$$\nwhere $\\theta’$ is a moving average of $\\theta$, which is a\nmean teacher\n.\nOverview of Interpolation Consistency Training. MixUp is applied to produce more interpolated samples with interpolated labels as learning targets. (Image source:\nVerma et al. 2019\n)\nBecause the probability of two randomly selected unlabeled samples belonging to different classes is high (e.g. There are 1000 object classes in ImageNet), the interpolation by applying a mixup between two random unlabeled samples is likely to happen around the decision boundary. According to the low-density separation\nassumptions\n, the decision boundary tends to locate in the low density regions.\n$$\n\\mathcal{L}^\\text{ICT}_{u} = \\mathbb{E}_{\\mathbf{u}_i, \\mathbf{u}_j \\sim \\mathcal{U}} \\mathbb{E}_{\\lambda \\sim \\text{Beta}(\\alpha, \\alpha)} D[p_\\theta(y \\mid \\text{mixup}_\\lambda (\\mathbf{u}_i, \\mathbf{u}_j)), \\text{mixup}_\\lambda(p_{\\theta’}(y \\mid \\mathbf{u}_i), p_{\\theta'}(y \\mid \\mathbf{u}_j)]\n$$\nwhere $\\theta’$ is a moving average of $\\theta$.\nSimilar to VAT,\nUnsupervised Data Augmentation\n(\nUDA\n;\nXie et al. 2020\n) learns to predict the same output for an unlabeled example and the augmented one. UDA especially focuses on studying how the\n“quality”\nof noise can impact the semi-supervised learning performance with consistency training. It is crucial to use advanced data augmentation methods for producing meaningful and effective noisy samples. Good data augmentation should produce valid (i.e. does not change the label) and diverse noise, and carry targeted inductive biases.\nFor images, UDA adopts RandAugment (\nCubuk et al. 2019\n) which uniformly samples augmentation operations available in\nPIL\n, no learning or optimization, so it is much cheaper than AutoAugment.\nComparison of various semi-supervised learning methods on CIFAR-10 classification. Fully supervised Wide-ResNet-28-2 and PyramidNet+ShakeDrop have an error rate of **5.4** and **2.7** respectively when trained on 50,000 examples without RandAugment. (Image source:\nXie et al. 2020\n)\nFor language, UDA combines back-translation and TF-IDF based word replacement. Back-translation preserves the high-level meaning but may not retain certain words, while TF-IDF based word replacement drops uninformative words with low TF-IDF scores. In the experiments on language tasks, they found UDA to be complementary to transfer learning and representation learning; For example, BERT fine-tuned (i.e. $\\text{BERT}_\\text{FINETUNE}$ in Fig. 8.) on in-domain unlabeled data can further improve the performance.\nComparison of UDA with different initialization configurations on various text classification tasks. (Image source:\nXie et al. 2020\n)\nWhen calculating $\\mathcal{L}_u$, UDA found two training techniques to help improve the results.\nLow confidence masking\n: Mask out examples with low prediction confidence if lower than a threshold $\\tau$.\nSharpening prediction distribution\n: Use a low temperature $T$ in softmax to sharpen the predicted probability distribution.\nIn-domain data filtration\n: In order to extract more in-domain data from a large out-of-domain dataset, they trained a classifier to predict in-domain labels and then retain samples with high confidence predictions as in-domain candidates.\n$$\n\\begin{aligned}\n&\\mathcal{L}_u^\\text{UDA} = \\mathbb{1}[\\max_{y'} p_{\\hat{\\theta}}(y'\\mid \\mathbf{x}) > \\tau ] \\cdot D[p^\\text{(sharp)}_{\\hat{\\theta}}(y \\mid \\mathbf{x}; T), p_\\theta(y \\mid \\bar{\\mathbf{x}})] \\\\\n&\\text{where } p_{\\hat{\\theta}}^\\text{(sharp)}(y \\mid \\mathbf{x}; T) = \\frac{\\exp(z^{(y)} / T)}{ \\sum_{y'} \\exp(z^{(y')} / T) }\n\\end{aligned}\n$$\nwhere $\\hat{\\theta}$ is a fixed copy of model weights, same as in VAT, so no gradient update, and $\\bar{\\mathbf{x}}$ is the augmented data point. $\\tau$ is the prediction confidence threshold and $T$ is the distribution sharpening temperature.\nPseudo Labeling\n#\nPseudo Labeling\n(\nLee 2013\n) assigns fake labels to unlabeled samples based on the maximum softmax probabilities predicted by the current model and then trains the model on both labeled and unlabeled samples simultaneously in a pure supervised setup.\nWhy could pseudo labels work? Pseudo label is in effect equivalent to\nEntropy Regularization\n(\nGrandvalet & Bengio 2004\n), which minimizes the conditional entropy of class probabilities for unlabeled data to favor low density separation between classes. In other words, the predicted class probabilities is in fact a measure of class overlap, minimizing the entropy is equivalent to reduced class overlap and thus low density separation.\nt-SNE visualization of outputs on MNIST test set by models training (a) without and (b) with pseudo labeling on 60000 unlabeled samples, in addition to 600 labeled data. Pseudo labeling leads to better segregation in the learned embedding space.  (Image source:\nLee 2013\n)\nTraining with pseudo labeling naturally comes as an iterative process. We refer to the model that produces pseudo labels as teacher and the model that learns with pseudo labels as student.\nLabel propagation\n#\nLabel Propagation\n(\nIscen et al. 2019\n) is an idea to construct a similarity graph among samples based on feature embedding. Then the pseudo labels are “diffused” from known samples to unlabeled ones where the propagation weights are proportional to pairwise similarity scores in the graph. Conceptually it is similar to a k-NN classifier and both suffer from the problem of not scaling up well with a large dataset.\nIllustration of how Label Propagation works. (Image source:\nIscen et al. 2019\n)\nSelf-Training\n#\nSelf-Training\nis not a new concept (\nScudder 1965\n,\nNigram & Ghani CIKM 2000\n). It is an iterative algorithm, alternating between the following two steps until every unlabeled sample has a label assigned:\nInitially it builds a classifier on labeled data.\nThen it uses this classifier to predict labels for the unlabeled data and converts the most confident ones into labeled samples.\nXie et al. (2020)\napplied self-training in deep learning and achieved great results. On the ImageNet classification task, they first trained an EfficientNet (\nTan & Le 2019\n) model as teacher to generate pseudo labels for 300M unlabeled images and then trained a larger EfficientNet as student to learn with both true labeled and pseudo labeled images. One critical element in their setup is to have\nnoise\nduring student model training but have no noise for the teacher to produce pseudo labels. Thus their method is called\nNoisy Student\n. They applied stochastic depth (\nHuang et al. 2016\n), dropout and RandAugment to noise the student. Noise is important for the student to perform better than the teacher. The added noise has a compound effect to encourage the model’s decision making frontier to be smooth, on both labeled and unlabeled data.\nA few other important technical configs in noisy student self-training are:\nThe student model should be sufficiently large (i.e. larger than the teacher) to fit more data.\nNoisy student should be paired with data balancing, especially important to balance the number of pseudo labeled images in each class.\nSoft pseudo labels work better than hard ones.\nNoisy student also improves adversarial robustness against an FGSM (Fast Gradient Sign Attack = The attack uses the gradient of the loss w.r.t the input data and adjusts the input data to maximize the loss) attack though the model is not optimized for adversarial robustness.\nSentAugment, proposed by\nDu et al. (2020)\n, aims to solve the problem when there is not enough in-domain unlabeled data for self-training in the language domain. It relies on sentence embedding to find unlabeled in-domain samples from a large corpus and uses the retrieved sentences for self-training.\nReducing confirmation bias\n#\nConfirmation bias is a problem with incorrect pseudo labels provided by an imperfect teacher model. Overfitting to wrong labels may not give us a better student model.\nTo reduce confirmation bias,\nArazo et al. (2019)\nproposed  two techniques. One is to adopt MixUp with soft labels. Given two samples, $(\\mathbf{x}_i, \\mathbf{x}_j)$ and their corresponding true or pseudo labels $(y_i, y_j)$, the interpolated label equation can be translated to a cross entropy loss with softmax outputs:\n$$\n\\begin{aligned}\n&\\bar{\\mathbf{x}} = \\lambda \\mathbf{x}_i + (1-\\lambda) \\mathbf{x}_j \\\\\n&\\bar{y} = \\lambda y_i + (1-\\lambda) y_j \\Leftrightarrow\n\\mathcal{L} = \\lambda [y_i^\\top \\log f_\\theta(\\bar{\\mathbf{x}})] + (1-\\lambda) [y_j^\\top \\log f_\\theta(\\bar{\\mathbf{x}})]\n\\end{aligned}\n$$\nMixup is insufficient if there are too few labeled samples. They further set a minimum number of labeled samples in every mini batch by oversampling the labeled samples. This works better than upweighting labeled samples, because it leads to more frequent updates rather than few updates of larger magnitude which could be less stable. Like consistency regularization, data augmentation and dropout are also important for pseudo labeling to work well.\nMeta Pseudo Labels\n(\nPham et al. 2021\n) adapts the teacher model constantly with the feedback of how well the student performs on the labeled dataset. The teacher and the student are trained in parallel, where the teacher learns to generate better pseudo labels and the student learns from the pseudo labels.\nLet the teacher and student model weights be $\\theta_T$ and $\\theta_S$, respectively. The student model’s loss on the labeled samples is defined as a function $\\theta^\\text{PL}_S(.)$ of $\\theta_T$ and we would like to minimize this loss by optimizing the teacher model accordingly.\n$$\n\\begin{aligned}\n\\min_{\\theta_T} &\\mathcal{L}_s(\\theta^\\text{PL}_S(\\theta_T)) = \\min_{\\theta_T} \\mathbb{E}_{(\\mathbf{x}^l, y) \\in \\mathcal{X}} \\text{CE}[y, f_{\\theta_S}(\\mathbf{x}^l)]  \\\\\n\\text{where } &\\theta^\\text{PL}_S(\\theta_T)\n= \\arg\\min_{\\theta_S} \\mathcal{L}_u (\\theta_T, \\theta_S)\n= \\arg\\min_{\\theta_S} \\mathbb{E}_{\\mathbf{u} \\sim \\mathcal{U}} \\text{CE}[(f_{\\theta_T}(\\mathbf{u}), f_{\\theta_S}(\\mathbf{u}))]\n\\end{aligned}\n$$\nHowever, it is not trivial to optimize the above equation. Borrowing the idea of\nMAML\n, it approximates the multi-step $\\arg\\min_{\\theta_S}$ with the one-step gradient update of $\\theta_S$,\n$$\n\\begin{aligned}\n\\theta^\\text{PL}_S(\\theta_T) &\\approx \\theta_S - \\eta_S \\cdot \\nabla_{\\theta_S} \\mathcal{L}_u(\\theta_T, \\theta_S) \\\\\n\\min_{\\theta_T} \\mathcal{L}_s (\\theta^\\text{PL}_S(\\theta_T)) &\\approx \\min_{\\theta_T} \\mathcal{L}_s \\big( \\theta_S - \\eta_S \\cdot \\nabla_{\\theta_S} \\mathcal{L}_u(\\theta_T, \\theta_S) \\big)\n\\end{aligned}\n$$\nWith soft pseudo labels, the above objective is differentiable. But if using hard pseudo labels, it is not differentiable and thus we need to use RL, e.g. REINFORCE.\nThe optimization procedure is alternative between training two models:\nStudent model update\n: Given a batch of unlabeled samples $\\{ \\mathbf{u} \\}$, we generate pseudo labels by $f_{\\theta_T}(\\mathbf{u})$ and optimize $\\theta_S$ with one step SGD: $\\theta’_S = \\color{green}{\\theta_S - \\eta_S \\cdot \\nabla_{\\theta_S} \\mathcal{L}_u(\\theta_T, \\theta_S)}$.\nTeacher model update\n: Given a batch of labeled samples $\\{(\\mathbf{x}^l, y)\\}$, we reuse the student’s update to optimize $\\theta_T$: $\\theta’_T = \\theta_T  - \\eta_T \\cdot \\nabla_{\\theta_T} \\mathcal{L}_s ( \\color{green}{\\theta_S - \\eta_S \\cdot \\nabla_{\\theta_S} \\mathcal{L}_u(\\theta_T, \\theta_S)} )$. In addition, the UDA objective is applied to the teacher model to incorporate consistency regularization.\nComparison of Meta Pseudo Labels with other semi- or self-supervised learning methods on image classification tasks. (Image source:\nPham et al. 2021\n)\nPseudo Labeling with Consistency Regularization\n#\nIt is possible to combine the above two approaches together, running semi-supervised learning with both pseudo labeling and consistency training.\nMixMatch\n#\nMixMatch\n(\nBerthelot et al. 2019\n), as a holistic approach to semi-supervised learning, utilizes unlabeled data by merging the following techniques:\nConsistency regularization\n: Encourage the model to output the same predictions on perturbed unlabeled samples.\nEntropy minimization\n: Encourage the model to output confident predictions on unlabeled data.\nMixUp\naugmentation: Encourage the model to have linear behaviour between samples.\nGiven a batch of labeled data $\\mathcal{X}$ and unlabeled data $\\mathcal{U}$, we create augmented versions of them via $\\text{MixMatch}(.)$, $\\bar{\\mathcal{X}}$ and $\\bar{\\mathcal{U}}$, containing augmented samples and guessed labels for unlabeled examples.\n$$\n\\begin{aligned}\n\\bar{\\mathcal{X}}, \\bar{\\mathcal{U}} &= \\text{MixMatch}(\\mathcal{X}, \\mathcal{U}, T, K, \\alpha) \\\\\n\\mathcal{L}^\\text{MM}_s &= \\frac{1}{\\vert \\bar{\\mathcal{X}} \\vert} \\sum_{(\\bar{\\mathbf{x}}^l, y)\\in \\bar{\\mathcal{X}}} D[y, p_\\theta(y \\mid \\bar{\\mathbf{x}}^l)] \\\\\n\\mathcal{L}^\\text{MM}_u &= \\frac{1}{L\\vert \\bar{\\mathcal{U}} \\vert} \\sum_{(\\bar{\\mathbf{u}}, \\hat{y})\\in \\bar{\\mathcal{U}}} \\| \\hat{y} - p_\\theta(y \\mid \\bar{\\mathbf{u}}) \\|^2_2 \\\\\n\\end{aligned}\n$$\nwhere $T$ is the sharpening temperature to reduce the guessed label overlap; $K$ is the number of augmentations generated per unlabeled example; $\\alpha$ is the parameter in MixUp.\nFor each $\\mathbf{u}$, MixMatch generates $K$ augmentations, $\\bar{\\mathbf{u}}^{(k)} = \\text{Augment}(\\mathbf{u})$ for $k=1, \\dots, K$ and the pseudo label is guessed based on the average: $\\hat{y} = \\frac{1}{K} \\sum_{k=1}^K p_\\theta(y \\mid \\bar{\\mathbf{u}}^{(k)})$.\nThe process of \"label guessing\" in MixMatch: averaging $K$ augmentations, correcting the predicted marginal distribution and finally sharpening the distribution. (Image source:\nBerthelot et al. 2019\n)\nAccording to their ablation studies, it is critical to have MixUp especially on the unlabeled data. Removing temperature sharpening on the pseudo label distribution hurts the performance quite a lot. Average over multiple augmentations for label guessing is also necessary.\nReMixMatch\n(\nBerthelot et al. 2020\n) improves MixMatch by introducing two new mechanisms:\nIllustration of two improvements introduced in ReMixMatch over MixMatch. (Image source:\nBerthelot et al. 2020\n)\nDistribution alignment.\nIt encourages the marginal distribution $p(y)$ to be close to the marginal distribution of the ground truth labels. Let $p(y)$ be the class distribution in the true labels and $\\tilde{p}(\\hat{y})$ be a running average of the predicted class distribution among the unlabeled data. The model prediction on an unlabeled sample $p_\\theta(y \\vert \\mathbf{u})$ is normalized to be $\\text{Normalize}\\big( \\frac{p_\\theta(y \\vert \\mathbf{u}) p(y)}{\\tilde{p}(\\hat{y})} \\big)$ to match the true marginal distribution.\nNote that entropy minimization is not a useful objective if the marginal distribution is not uniform.\nI do feel the assumption that the class distributions on the labeled and unlabeled data should match is too strong and not necessarily to be true in the real-world setting.\nAugmentation anchoring\n. Given an unlabeled sample, it first generates an “anchor” version with weak augmentation and then averages $K$ strongly augmented versions using CTAugment (Control Theory Augment). CTAugment only samples augmentations that keep the model predictions within the network tolerance.\nThe ReMixMatch loss is a combination of several terms,\na supervised loss with data augmentation and MixUp applied;\nan unsupervised loss with data augmentation and MixUp applied, using pseudo labels as targets;\na CE loss on a single heavily-augmented unlabeled image without MixUp;\na\nrotation\nloss as in self-supervised learning.\nDivideMix\n#\nDivideMix\n(\nJunnan Li et al. 2020\n) combines semi-supervised learning with Learning with noisy labels (LNL). It models the per-sample loss distribution via a\nGMM\nto dynamically divide the training data into a labeled set with clean examples and an unlabeled set with noisy ones. Following the idea in\nArazo et al. 2019\n, they fit a two-component GMM on the per-sample cross entropy loss $\\ell_i = y_i^\\top \\log f_\\theta(\\mathbf{x}_i)$. Clean samples are expected to get lower loss faster than noisy samples. The component with smaller mean is the cluster corresponding to clean labels and let’s denote it as $c$. If the GMM posterior probability $w_i = p_\\text{GMM}(c \\mid \\ell_i)$ (i.e. the probability of the sampling belonging to the clean sample set) is larger than the threshold $\\tau$, this sample is considered as a clean sample and otherwise a noisy one.\nThe data clustering step is named\nco-divide\n. To avoid confirmation bias, DivideMix simultaneously trains two diverged networks where each network uses the dataset division from the other network; e.g. thinking about how Double Q Learning works.\nDivideMix trains two networks independently to reduce confirmation bias. They run co-divide, co-refinement, and co-guessing together. (Image source:\nJunnan Li et al. 2020\n)\nCompared to MixMatch, DivideMix has an additional\nco-divide\nstage for handling noisy samples, as well as the following improvements during training:\nLabel co-refinement\n: It linearly combines the ground-truth label $y_i$ with the network’s prediction $\\hat{y}_i$, which is averaged across multiple augmentations of $\\mathbf{x}_i$, guided by the clean set probability $w_i$ produced by the other network.\nLabel co-guessing\n: It averages the predictions from two models for unlabelled data samples.\nThe algorithm of DivideMix. (Image source:\nJunnan Li et al. 2020\n)\nFixMatch\n#\nFixMatch\n(\nSohn et al. 2020\n) generates pseudo labels on unlabeled samples with weak augmentation and only keeps predictions with high confidence. Here both weak augmentation and high confidence filtering help produce high-quality trustworthy pseudo label targets. Then FixMatch learns to predict these pseudo labels given a heavily-augmented sample.\nIllustration of how FixMatch works. (Image source:\nSohn et al. 2020\n)\n$$\n\\begin{aligned}\n\\mathcal{L}_s &= \\frac{1}{B} \\sum^B_{b=1} \\text{CE}[y_b, p_\\theta(y \\mid \\mathcal{A}_\\text{weak}(\\mathbf{x}_b))] \\\\\n\\mathcal{L}_u &= \\frac{1}{\\mu B} \\sum_{b=1}^{\\mu B} \\mathbb{1}[\\max(\\hat{y}_b) \\geq \\tau]\\;\\text{CE}(\\hat{y}_b, p_\\theta(y \\mid \\mathcal{A}_\\text{strong}(\\mathbf{u}_b)))\n\\end{aligned}\n$$\nwhere $\\hat{y}_b$ is the pseudo label for an unlabeled example; $\\mu$ is a hyperparameter that determines the relative sizes of $\\mathcal{X}$ and $\\mathcal{U}$.\nWeak augmentation $\\mathcal{A}_\\text{weak}(.)$: A standard flip-and-shift augmentation\nStrong augmentation $\\mathcal{A}_\\text{strong}(.)$ : AutoAugment, Cutout, RandAugment, CTAugment\nPerformance of FixMatch and several other semi-supervised learning methods on image classification tasks. (Image source:\nSohn et al. 2020\n)\nAccording to the ablation studies of FixMatch,\nSharpening the predicted distribution with a temperature parameter $T$ does not have a significant impact when the threshold $\\tau$ is used.\nCutout and CTAugment as part of strong augmentations are necessary for good performance.\nWhen the weak augmentation for label guessing is replaced with strong augmentation, the model diverges early in training. If discarding weak augmentation completely, the model overfit the guessed labels.\nUsing weak instead of strong augmentation for pseudo label prediction leads to unstable performance. Strong data augmentation is critical.\nCombined with Powerful Pre-Training\n#\nIt is a common paradigm, especially in language tasks, to first pre-train a task-agnostic model on a large unsupervised data corpus via self-supervised learning and then fine-tune it on the downstream task with a small labeled dataset. Research has shown that we can obtain extra gain if combining semi-supervised learning with pretraining.\nZoph et al. (2020)\nstudied to what degree\nself-training\ncan work better than pre-training. Their experiment setup was to use ImageNet for pre-training or self-training to improve COCO. Note that when using ImageNet for self-training, it discards labels and only uses ImageNet samples as unlabeled data points.\nHe et al. (2018)\nhas demonstrated that ImageNet classification pre-training does not work well if the downstream task is very different, such as object detection.\nThe effect of (a) data augment (from weak to strong) and (b) the labeled dataset size on the object detection performance. In the legend: `Rand Init` refers to a model initialized w/ random weights; `ImageNet` is initialized with a pre-trained checkpoint at 84.5% top-1 ImageNet accuracy; `ImageNet++` is initialized with a checkpoint with a higher accuracy 86.9%. (Image source:\nZoph et al. 2020\n)\nTheir experiments demonstrated a series of interesting findings:\nThe effectiveness of pre-training diminishes with more labeled samples available for the downstream task. Pre-training is helpful in the low-data regimes (20%) but neutral or harmful in the high-data regime.\nSelf-training helps in high data/strong augmentation regimes, even when pre-training hurts.\nSelf-training can bring in additive improvement on top of pre-training, even using the same data source.\nSelf-supervised pre-training (e.g. via SimCLR) hurts the performance in a high data regime, similar to how supervised pre-training does.\nJoint-training supervised and self-supervised objectives help resolve the mismatch between the pre-training and downstream tasks. Pre-training, joint-training and self-training are all additive.\nNoisy labels or un-targeted labeling (i.e. pre-training labels are not aligned with downstream task labels) is worse than targeted pseudo labeling.\nSelf-training is computationally more expensive than fine-tuning on a pre-trained model.\nChen et al. (2020)\nproposed a three-step procedure to merge the benefits of self-supervised pretraining, supervised fine-tuning and self-training together:\nUnsupervised or self-supervised pretrain a big model.\nSupervised fine-tune it on a few labeled examples. It is important to use a big (deep and wide) neural network.\nBigger models yield better performance with fewer labeled samples.\nDistillation with unlabeled examples by adopting pseudo labels in self-training.\nIt is possible to distill the knowledge from a large model into a small one because the task-specific use does not require extra capacity of the learned representation.\nThe distillation loss is formatted as the following, where the teacher network is fixed with weights $\\hat{\\theta}_T$.\n$$\n     \\mathcal{L}_\\text{distill} = - (1-\\alpha) \\underbrace{\\sum_{(\\mathbf{x}^l_i, y_i) \\in \\mathcal{X}} \\big[ \\log p_{\\theta_S}(y_i \\mid \\mathbf{x}^l_i) \\big]}_\\text{Supervised loss} - \\alpha \\underbrace{\\sum_{\\mathbf{u}_i \\in \\mathcal{U}} \\Big[ \\sum_{i=1}^L p_{\\hat{\\theta}_T}(y^{(i)} \\mid \\mathbf{u}_i; T) \\log p_{\\theta_S}(y^{(i)} \\mid \\mathbf{u}_i; T) \\Big]}_\\text{Distillation loss using unlabeled data}\n     $$\nA semi-supervised learning framework leverages unlabeled data corpus by (Left) task-agnostic unsupervised pretraining and (Right) task-specific self-training and distillation. (Image source:\nChen et al. 2020\n)\nThey experimented on the ImageNet classification task. The self-supervised pre-training uses SimCLRv2, a directly improved version of\nSimCLR\n. Observations in their empirical studies confirmed several learnings, aligned with\nZoph et al. 2020\n:\nBigger models are more label-efficient;\nBigger/deeper project heads in SimCLR improve representation learning;\nDistillation using unlabeled data improves semi-supervised learning.\nComparison of performance by SimCLRv2 + semi-supervised distillation on ImageNet classification. (Image source:\nChen et al. 2020\n)\n💡 Quick summary of common themes among recent semi-supervised learning methods, many aiming to reduce confirmation bias:\nApply valid and diverse noise to samples by advanced data augmentation methods.\nWhen dealing with images, MixUp is an effective augmentation. Mixup could work on language too, resulting in a small incremental improvement (\nGuo et al. 2019\n).\nSet a threshold and discard pseudo labels with low confidence.\nSet a minimum number of labeled samples per mini-batch.\nSharpen the pseudo label distribution to reduce the class overlap.\nCitation\n#\nCited as:\nWeng, Lilian. (Dec 2021). Learning with not enough data part 1: semi-supervised learning. Lil’Log. https://lilianweng.github.io/posts/2021-12-05-semi-supervised/.\nOr\n@article{weng2021semi,\n  title   = \"Learning with not Enough Data Part 1: Semi-Supervised Learning\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2021\",\n  month   = \"Dec\",\n  url     = \"https://lilianweng.github.io/posts/2021-12-05-semi-supervised/\"\n}\nReferences\n#\n[1] Ouali, Hudelot & Tami.\n“An Overview of Deep Semi-Supervised Learning”\narXiv preprint arXiv:2006.05278 (2020).\n[2] Sajjadi, Javanmardi & Tasdizen\n“Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning.”\narXiv preprint arXiv:1606.04586 (2016).\n[3] Pham et al.\n“Meta Pseudo Labels.”\nCVPR 2021.\n[4] Laine & Aila.\n“Temporal Ensembling for Semi-Supervised Learning”\nICLR 2017.\n[5] Tarvaninen & Valpola.\n“Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.”\nNeuriPS 2017\n[6] Xie et al.\n“Unsupervised Data Augmentation for Consistency Training.”\nNeuriPS 2020.\n[7] Miyato et al.\n“Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning.”\nIEEE transactions on pattern analysis and machine intelligence 41.8 (2018).\n[8] Verma et al.\n“Interpolation consistency training for semi-supervised learning.”\nIJCAI 2019\n[9] Lee.\n“Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.”\nICML 2013 Workshop: Challenges in Representation Learning.\n[10] Iscen et al.\n“Label propagation for deep semi-supervised learning.”\nCVPR 2019.\n[11] Xie et al.\n“Self-training with Noisy Student improves ImageNet classification”\nCVPR 2020.\n[12] Jingfei Du et al.\n“Self-training Improves Pre-training for Natural Language Understanding.”\n2020\n[13] Iscen et al.\n“Label propagation for deep semi-supervised learning.”\nCVPR 2019\n[14] Arazo et al.\n“Pseudo-labeling and confirmation bias in deep semi-supervised learning.”\nIJCNN 2020.\n[15] Berthelot et al.\n“MixMatch: A holistic approach to semi-supervised learning.”\nNeuriPS 2019\n[16] Berthelot et al.\n“ReMixMatch: Semi-supervised learning with distribution alignment and augmentation anchoring.”\nICLR 2020\n[17] Sohn et al.\n“FixMatch: Simplifying semi-supervised learning with consistency and confidence.”\nCVPR 2020\n[18] Junnan Li et al.\n“DivideMix: Learning with Noisy Labels as Semi-supervised Learning.”\n2020 [\ncode\n]\n[19] Zoph et al.\n“Rethinking pre-training and self-training.”\n2020.\n[20] Chen et al.\n“Big Self-Supervised Models are Strong Semi-Supervised Learners”\n2020",
  "metadata": {
    "title": "Learning with not Enough Data Part 1: Semi-Supervised Learning | Lil'Log",
    "description": "When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.",
    "keywords": "data, unsupervised-learning",
    "author": "Lilian Weng",
    "og:title": "Learning with not Enough Data Part 1: Semi-Supervised Learning",
    "og:description": "When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/",
    "twitter:card": "summary",
    "twitter:title": "Learning with not Enough Data Part 1: Semi-Supervised Learning",
    "twitter:description": "When facing a limited amount of labeled data for supervised learning tasks, four approaches are commonly discussed.",
    "article:section": "posts",
    "article:published_time": "2021-12-05T00:00:00+00:00",
    "article:modified_time": "2021-12-05T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/76583c4985bd/code/body-000.txt",
      "chars": 283,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/PI-model.png",
      "bytes": 77905,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-22f8744911da963ff507.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/PI-model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/temperal-ensembling.png",
      "bytes": 83063,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-2431b04e51e95d44cbfc.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/temperal-ensembling.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/mean-teacher.png",
      "bytes": 147122,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-3704ca968b8b036dd979.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/mean-teacher.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/mean-teacher-results.png",
      "bytes": 449224,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-264593afb504dba40862.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/mean-teacher-results.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/consistency-training-with-noisy-samples.png",
      "bytes": 270328,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-fe1f57c36a0c6b35ff30.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/consistency-training-with-noisy-samples.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/ICT.png",
      "bytes": 299310,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-686784e7314725fbdcc3.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/ICT.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/UDA-image-results.png",
      "bytes": 375554,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-abc3c1e68e088b944f70.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/UDA-image-results.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/UDA-language-results.png",
      "bytes": 278212,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-20f5ad7f538bcfcaa51f.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/UDA-language-results.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/pseudo-label-segregation.png",
      "bytes": 3492681,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-09be84ee222f7edd8341.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/pseudo-label-segregation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/label-propagation.png",
      "bytes": 340862,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-41918aaaac4a38312f21.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/label-propagation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/MPL-results.png",
      "bytes": 373018,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-151ce877219f628d821e.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/MPL-results.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/MixMatch.png",
      "bytes": 130110,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-52dcd8696c072fec5989.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/MixMatch.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/ReMixMatch.png",
      "bytes": 519536,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-60ae1da08ae38b4e5477.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/ReMixMatch.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/DivideMix.png",
      "bytes": 124124,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-1f3e79d5973995850e17.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/DivideMix.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/DivideMix-algo.png",
      "bytes": 548196,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-b8bdf2d20d1fcc25c810.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/DivideMix-algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/FixMatch.png",
      "bytes": 218342,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-015dfefe2091cea2d5d5.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/FixMatch.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/FixMatch-results.png",
      "bytes": 306334,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-de572b70c4267a43c66a.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/FixMatch-results.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/self-training-pre-training.png",
      "bytes": 470728,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-cf42d45d1104e492f130.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/self-training-pre-training.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/big-self-supervised-model.png",
      "bytes": 176722,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-452ce3c3d7e9a8712c06.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/big-self-supervised-model.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/big-self-supervised-model-results.png",
      "bytes": 411449,
      "reused": false,
      "local_path": "attachments/76583c4985bd/images/body-9d2b0d8db4abe7b366db.png",
      "src": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/big-self-supervised-model-results.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://lilianweng.github.io/posts/2019-01-31-lm/",
      "label": "pre-training LMs",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2019-11-10-self-supervised/",
      "label": "self-supervised learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.05278",
      "label": "this survey",
      "type": "a"
    },
    {
      "href": "https://stats.stackexchange.com/questions/66939/what-is-the-manifold-assumption-in-semi-supervised-learning",
      "label": "a helpful link",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2021-05-31-contrastive/",
      "label": "learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1610.02242",
      "label": "Laine & Aila (2017)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1606.04586",
      "label": "Sajjadi et al. (2016)",
      "type": "a"
    },
    {
      "href": "https://stats.stackexchange.com/questions/232741/why-is-it-important-to-include-a-bias-correction-term-for-the-adam-optimizer-for",
      "label": "bias correction",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1703.01780",
      "label": "Tarvaninen & Valpola, 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1412.6572",
      "label": "Goodfellow et al. 2014",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1704.03976",
      "label": "Miyato et al. 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1903.03825",
      "label": "Verma et al. 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1710.09412",
      "label": "Zheng et al. 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.12848",
      "label": "Xie et al. 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1909.13719",
      "label": "Cubuk et al. 2019",
      "type": "a"
    },
    {
      "href": "https://pillow.readthedocs.io/en/stable/",
      "label": "PIL",
      "type": "a"
    },
    {
      "href": "http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.664.3543&rep=rep1&type=pdf",
      "label": "Lee 2013",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/2004/hash/96f2b50b5d3613adf9c27049b2a888c7-Abstract.html",
      "label": "Grandvalet & Bengio 2004",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.04717",
      "label": "Iscen et al. 2019",
      "type": "a"
    },
    {
      "href": "https://ieeexplore.ieee.org/document/1053799",
      "label": "Scudder 1965",
      "type": "a"
    },
    {
      "href": "http://www.kamalnigam.com/papers/cotrain-CIKM00.pdf",
      "label": "Nigram & Ghani CIKM 2000",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.04252",
      "label": "Xie et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.11946",
      "label": "Tan & Le 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1603.09382",
      "label": "Huang et al. 2016",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.02194",
      "label": "Du et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1908.02983",
      "label": "Arazo et al. (2019)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2003.10580",
      "label": "Pham et al. 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1703.03400",
      "label": "MAML",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.02249",
      "label": "Berthelot et al. 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.09785",
      "label": "Berthelot et al. 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2002.07394",
      "label": "Junnan Li et al. 2020",
      "type": "a"
    },
    {
      "href": "https://scikit-learn.org/stable/modules/mixture.html",
      "label": "GMM",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.11238",
      "label": "Arazo et al. 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2001.07685",
      "label": "Sohn et al. 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.06882",
      "label": "Zoph et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1811.08883",
      "label": "He et al. (2018)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.10029",
      "label": "Chen et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.08941",
      "label": "Guo et al. 2019",
      "type": "a"
    },
    {
      "href": "https://github.com/LiJunnan1992/DivideMix",
      "label": "code",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:21Z",
  "source_url": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.05278",
      "bytes": 3172260,
      "reused": false,
      "local_path": "papers/repair/85bc9fdf7b6d2f5c5ff6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1610.02242",
      "bytes": 322144,
      "reused": false,
      "local_path": "papers/repair/1366267456ae3965f8c9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1606.04586",
      "bytes": 282460,
      "reused": false,
      "local_path": "papers/repair/dfcdfea8469d7c081d06.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.01780",
      "bytes": 863822,
      "reused": false,
      "local_path": "papers/repair/f5e8782daef1aa339f71.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1412.6572",
      "bytes": 1036360,
      "reused": false,
      "local_path": "papers/repair/1b6f12d7db56a03ce596.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1704.03976",
      "bytes": 8496008,
      "reused": false,
      "local_path": "papers/repair/7c610a3bc37a3c724c6c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1903.03825",
      "bytes": 2774709,
      "reused": false,
      "local_path": "papers/repair/c0ed786bf3a3169d24a7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1710.09412",
      "bytes": 740562,
      "reused": false,
      "local_path": "papers/repair/7af2d52fbf38a69ec251.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.12848",
      "bytes": 1625724,
      "reused": false,
      "local_path": "papers/repair/17af125202b45606d9ad.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1909.13719",
      "bytes": 1263406,
      "reused": false,
      "local_path": "papers/repair/c267ff367c4770ddb6c0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.04717",
      "bytes": 626230,
      "reused": false,
      "local_path": "papers/repair/ae6e8a0596cf6995095d.pdf"
    },
    {
      "status": "ok",
      "url": "http://www.kamalnigam.com/papers/cotrain-CIKM00.pdf",
      "bytes": 230483,
      "reused": false,
      "local_path": "papers/repair/aa557a37f131c5edaca1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.04252",
      "bytes": 2801395,
      "reused": false,
      "local_path": "papers/repair/2d5071bca517ed21302d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.11946",
      "bytes": 933600,
      "reused": false,
      "local_path": "papers/repair/b9c9141c18b68865cdc7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1603.09382",
      "bytes": 989025,
      "reused": false,
      "local_path": "papers/repair/531ae1708a14ac8b81dc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.02194",
      "bytes": 2525939,
      "reused": false,
      "local_path": "papers/repair/46cf1aae6a3b935b22db.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1908.02983",
      "bytes": 904317,
      "reused": false,
      "local_path": "papers/repair/4e39b8b3af735acbdef2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2003.10580",
      "bytes": 1245388,
      "reused": false,
      "local_path": "papers/repair/19c860b2949c2d91d5ec.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.03400",
      "bytes": 4064212,
      "reused": false,
      "local_path": "papers/repair/745c6af0eff74ad13278.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.02249",
      "bytes": 520905,
      "reused": false,
      "local_path": "papers/repair/c84670ad345d150685e0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.09785",
      "bytes": 420212,
      "reused": false,
      "local_path": "papers/repair/e1bda50022ee2a740e73.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2002.07394",
      "bytes": 771769,
      "reused": false,
      "local_path": "papers/repair/a688eaa44d9b3d4d3846.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.11238",
      "bytes": 3862031,
      "reused": false,
      "local_path": "papers/repair/4f8506d9f9d2559e3846.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2001.07685",
      "bytes": 770166,
      "reused": false,
      "local_path": "papers/repair/f068f1c17b46bc3eed3a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.06882",
      "bytes": 1655938,
      "reused": false,
      "local_path": "papers/repair/d7e01a0318a00ae28138.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1811.08883",
      "bytes": 788571,
      "reused": false,
      "local_path": "papers/repair/db0e3c5521e5a1800c77.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.10029",
      "bytes": 823706,
      "reused": false,
      "local_path": "papers/repair/558bd6a836d462b86d7a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.08941",
      "bytes": 345640,
      "reused": true,
      "local_path": "papers/repair/646a76cd251b8253359a.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 39098,
  "expected_body_images": 20,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/76583c4985bd/pages/source.html",
  "local_html": "attachments/76583c4985bd/pages/article.html",
  "body_base_path": "attachments/76583c4985bd/pages/",
  "body_markdown_path": "attachments/76583c4985bd/pages/article.md"
}