{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "Learning with not Enough Data Part 2: Active Learning",
  "keywords": "主动学习,数据标注,采样",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/",
  "publish_date": "2022-02",
  "corpus_stage": "数据处理",
  "notes": "主动学习系列综述之二",
  "id": "028a8389cdec",
  "collected_at": "2026-10-02T14:13:03",
  "title": "Learning with not Enough Data Part 2: Active Learning | Lil'Log",
  "body_markdown": "This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.\nNotations\n#\nSymbol\nMeaning\n$K$\nNumber of unique class labels.\n$(\\mathbf{x}^l, y) \\sim \\mathcal{X}, y \\in \\{0, 1\\}^K$\nLabeled dataset. $y$ is a one-hot representation of the true label.\n$\\mathbf{u} \\sim \\mathcal{U}$\nUnlabeled dataset.\n$\\mathcal{D} = \\mathcal{X} \\cup \\mathcal{U}$\nThe entire dataset, including both labeled and unlabeled examples.\n$\\mathbf{x}$\nAny sample which can be either labeled or unlabeled.\n$\\mathbf{x}_i$\nThe $i$-th sample.\n$U(\\mathbf{x})$\nScoring function for active learning selection.\n$P_\\theta(y \\vert \\mathbf{x})$\nA softmax classifier parameterized by $\\theta$.\n$\\hat{y} = \\arg\\max_{y \\in \\mathcal{Y}} P_\\theta(y \\vert \\mathbf{x})$\nThe most confident prediction by the classifier.\n$B$\nLabeling budget (the maximum number of samples to label).\n$b$\nBatch size.\nWhat is Active Learning?\n#\nGiven an unlabeled dataset $\\mathcal{U}$ and a fixed amount of labeling cost $B$, active learning aims to select a subset of $B$ examples from $\\mathcal{U}$ to be labeled such that they can result in maximized improvement in model performance. This is an effective way of learning especially when data labeling is difficult and costly, e.g. medical images. This classical\nsurvey paper\nin 2010 lists many key concepts. While some conventional approaches may not apply to deep learning, discussion in this post mainly focuses on deep neural models and training in batch mode.\nIllustration of a cyclic workflow of active learning, producing better models more efficiently by smartly choosing which samples to label.\nTo simplify the discussion, we assume that the task is a $K$-class classification problem in all the following sections. The model with parameters $\\theta$ outputs a probability distribution over the label candidates, which may or may not be calibrated, $P_\\theta(y \\vert \\mathbf{x})$ and the most likely prediction is $\\hat{y} = \\arg\\max_{y \\in \\mathcal{Y}} P_\\theta(y \\vert \\mathbf{x})$.\nAcquisition Function\n#\nThe process of identifying the most valuable examples to label next is referred to as “sampling strategy” or “query strategy”. The scoring function in the sampling process is named “acquisition function”, denoted as $U(\\mathbf{x})$. Data points with higher scores are expected to produce higher value for model training if they get labeled.\nHere is a list of basic sampling strategies.\nUncertainty Sampling\n#\nUncertainty sampling\nselects examples for which the model produces most uncertain predictions. Given a single model, uncertainty can be estimated by the predicted probabilities, although one common complaint is that deep learning model predictions are often not calibrated and not correlated with true uncertainty well. In fact, deep learning models are often overconfident.\nLeast confident score\n, also known as\nvariation ratio\n: $U(\\mathbf{x}) = 1 - P_\\theta(\\hat{y} \\vert \\mathbf{x})$.\nMargin score\n: $U(\\mathbf{x}) = P_\\theta(\\hat{y}_1 \\vert \\mathbf{x}) - P_\\theta(\\hat{y}_2 \\vert \\mathbf{x})$, where $\\hat{y}_1$ and $\\hat{y}_2$ are the most likely and the second likely predicted labels.\nEntropy\n: $U(\\mathbf{x}) = \\mathcal{H}(P_\\theta(y \\vert \\mathbf{x})) = - \\sum_{y \\in \\mathcal{Y}} P_\\theta(y \\vert \\mathbf{x}) \\log P_\\theta(y \\vert \\mathbf{x})$.\nAnother way to quantify uncertainty is to rely on a committee of expert models, known as Query-By-Committee (QBC). QBC measures uncertainty based on a pool of opinions and thus it is critical to keep a level of disagreement among committee members. Given $C$ models in the committee pool, each parameterized by $\\theta_1, \\dots, \\theta_C$.\nVoter entropy\n: $U(\\mathbf{x}) = \\mathcal{H}(\\frac{V(y)}{C})$, where $V(y)$ counts the number of votes from the committee on the label $y$.\nConsensus entropy\n: $U(\\mathbf{x}) = \\mathcal{H}(P_\\mathcal{C})$, where $P_\\mathcal{C}$ is the prediction averaging across the committee.\nKL divergence\n: $U(\\mathbf{x}) = \\frac{1}{C} \\sum_{c=1}^C D_\\text{KL} (P_{\\theta_c} | P_\\mathcal{C})$\nDiversity Sampling\n#\nDiversity sampling\nintend to find a collection of samples that can well represent the entire data distribution. Diversity is important because the model is expected to work well on any data in the wild, just not on a narrow subset. Selected samples should be representative of the underlying distribution. Common approaches often rely on quantifying the similarity between samples.\nExpected Model Change\n#\nExpected model change\nrefers to the impact that a sample brings onto the model training. The impact can be the influence on the model weights or the improvement over the training loss. A\nlater section\nreviews several works on how to measure model impact triggered by selected data samples.\nHybrid Strategy\n#\nMany methods above are not mutually exclusive. A\nhybrid\nsampling strategy values different attributes of data points, combining different sampling preferences into one. Often we want to select\nuncertain but also highly representative\nsamples.\nDeep Acquisition Function\n#\nMeasuring Uncertainty\n#\nThe model uncertainty is commonly categorized into two buckets (\nDer Kiureghian & Ditlevsen 2009\n,\nKendall & Gal 2017\n):\nAleatoric uncertainty\nis introduced by noise in the data (e.g. sensor data, noise in the measurement process) and it can be input-dependent or input-independent. It is generally considered as irreducible since there is missing information about the ground truth.\nEpistemic uncertainty\nrefers to the uncertainty within the model parameters and therefore we do not know whether the model can best explain the data. This type of uncertainty is theoretically reducible given more data\nEnsemble and Approximated Ensemble\n#\nThere is a long tradition in machine learning of using ensembles to improve model performance. When there is a significant diversity among models, ensembles are expected to yield better results. This ensemble theory is proved to be correct by many ML algorithms; for example,\nAdaBoost\naggregates many weak learners to perform similar or even better than a single strong learner.\nBootstrapping\nensembles multiple trials of resampling to achieve more accurate estimation of metrics. Random forests or\nGBM\nis also a good example for the effectiveness of ensembling.\nTo get better uncertainty estimation, it is intuitive to aggregate a collection of independently trained models. However, it is expensive to train a single deep neural network model, let alone many of them. In reinforcement learning, Bootstrapped DQN  (\nOsband, et al. 2016\n) is equipped with multiple value heads and relies on the uncertainty among an ensemble of Q value approximation to guide\nexploration\nin RL.\nIn active learning, a commoner approach is to use\ndropout\nto “simulate” a probabilistic Gaussian process (\nGal & Ghahramani 2016\n). We thus ensemble multiple samples collected from the same model but with different dropout masks applied during the forward pass to estimate the model uncertainty (epistemic uncertainty). The process is named\nMC dropout\n(Monte Carlo dropout), where dropout is applied before every weight layer, is approved to be mathematically equivalent to an approximation to the probabilistic deep Gaussian process (\nGal & Ghahramani 2016\n). This simple idea has been shown to be effective for classification with small datasets and widely adopted in scenarios when efficient model uncertainty estimation is needed.\nDBAL\n(Deep Bayesian active learning;\nGal et al. 2017\n) approximates Bayesian neural networks with MC dropout such that it learns a distribution over model weights. In their experiment, MC dropout performed better than random baseline and mean standard deviation (Mean STD), similarly to variation ratios and entropy measurement.\nActive learning results of DBAL on MNIST. (Image source:\nGal et al. 2017\n).\nBeluch et al. (2018)\ncompared ensemble-based models with MC dropout and found that the combination of naive ensemble (i.e. train multiple models separately and independently) and variation ratio yields better calibrated predictions than others. However, naive ensembles are\nvery\nexpensive, so they explored a few alternative cheaper options:\nSnapshot ensemble: Use a cyclic learning rate schedule to train an implicit ensemble such that it converges to different local minima.\nDiversity encouraging ensemble (DEE): Use a base network trained for a small number of epochs as initialization for $n$ different networks, each trained with dropout to encourage diversity.\nSplit head approach: One base model has multiple heads, each corresponding to one classifier.\nUnfortunately all the cheap implicit ensemble options above perform worse than naive ensembles. Considering the limit on computational resources, MC dropout is still a pretty good and economical choice. Naturally, people also try to combine ensemble and MC dropout (\nPop & Fulop 2018\n) to get a bit of additional performance gain by stochastic ensemble.\nUncertainty in Parameter Space\n#\nBayes-by-backprop\n(\nBlundell et al. 2015\n) measures weight uncertainty in neural networks directly. The method maintains a probability distribution over the weights $\\mathbf{w}$, which is modeled as a variational distribution $q(\\mathbf{w} \\vert \\theta)$ since the true posterior $p(\\mathbf{w} \\vert \\mathcal{D})$ is not tractable directly. The loss is to minimize the KL divergence between $q(\\mathbf{w} \\vert \\theta)$ and $p(\\mathbf{w} \\vert \\mathcal{D})$,\n$$\n\\begin{aligned}\n\\mathcal{L}(\\theta)\n&= \\text{KL}[q(\\mathbf{w}\\vert\\theta) \\| p(\\mathbf{w} \\vert \\mathcal{D})] \\\\ \n&= \\int q(\\mathbf{w}\\vert\\theta) \\log \\frac{q(\\mathbf{w}\\vert\\theta)}{p(\\mathbf{w}) p(\\mathcal{D}\\vert \\mathbf{w})} d\\mathbf{w} \\\\ \n&= \\text{KL}[q(\\mathbf{w}\\vert\\theta) \\| p(w)] - \\mathbb{E}_{q(\\mathbf{w}\\vert\\theta)} [\\log p(\\mathcal{D} \\vert \\mathbf{w})] \\\\\n&\\approx \\log q(\\mathbf{w} \\vert \\theta) - \\log p(\\mathbf{w}) p(\\mathcal{D}\\vert \\mathbf{w}) & \\text{; monte carlo sampling; }q(\\mathbf{w} \\vert \\theta)\\text{ & }p(\\mathbf{w})\\text{ are close.}\n\\end{aligned}\n$$\nThe variational distribution $q$ is typically a Gaussian with diagonal covariance and each weight is sampled from $\\mathcal{N}(\\mu_i, \\sigma_i^2)$. To ensure non-negativity of $\\sigma_i$, it is further parameterized via softplus, $\\sigma_i = \\log(1 + \\exp(\\rho_i))$ where the variational parameters are $\\theta = \\{\\mu_i , \\rho_i\\}^d_{i=1}$.\nThe process of Bayes-by-backprop can be summarized as:\nSample $\\epsilon \\sim \\mathcal{N}(0, I)$\nLet $\\mathbf{w} = \\mu + \\log(1+ \\exp(\\rho)) \\circ \\epsilon$\nLet $\\theta = (\\mu, \\rho)$\nLet $f(\\mathbf{w}, \\theta) = \\log q(\\mathbf{w} \\vert \\theta) - \\log p(\\mathbf{w})p(\\mathcal{D}\\vert \\mathbf{w})$\nCalculate the gradient of $f(\\mathbf{w}, \\theta)$ w.r.t. to $\\mu$ and $\\rho$ and then update $\\theta$.\nUncertainty is measured by sampling different model weights during inference.\nLoss Prediction\n#\nThe loss objective guides model training. A low loss value indicates that a model can make good and accurate predictions.\nYoo & Kweon (2019)\ndesigned a\nloss prediction module\nto predict the loss value for unlabeled inputs, as an estimation of how good a model prediction is on the given data. Data samples are selected if the loss prediction module makes uncertain predictions (high loss value) for them. The loss prediction module is a simple MLP with dropout, that takes several intermediate layer features as inputs and concatenates them after a global average pooling.\nUse the model with a loss prediction module to do active learning selection. (Image source:\nYoo & Kweon 2019\n)\nLet $\\hat{l}$ be the output of the loss prediction module and $l$ be the true loss. When training the loss prediction module, a simple MSE loss $=(l - \\hat{l})^2$ is not a good choice, because the loss decreases in time as the model learns to behave better. A good learning objective should be independent of the scale changes of the target loss. They instead rely on the comparison of sample pairs. Within each batch of size $b$, there are $b/2$ pairs of samples $(\\mathbf{x}_i, \\mathbf{x}_j)$ and the loss prediction model is expected to correctly predict which sample has a larger loss.\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{loss}(\\mathbf{x}_i, \\mathbf{x}_j) &= \\max\\big( 0, -\\mathbb{1}(l(\\mathbf{x}_i), l(\\mathbf{x}_j)) \\cdot (\\hat{l}(\\mathbf{x}_i) - \\hat{l}(\\mathbf{x}_j)) + \\epsilon \\big) \\\\ \n\\text{where } \\mathbb{1}(l_i, l_j) &= \\begin{cases} +1 & \\text{if }l_i > l_j \\\\ -1 & \\text{otherwise} \\end{cases} \n\\end{aligned}\n$$\nwhere $\\epsilon$ is a predefined positive margin constant.\nIn experiments on three vision tasks, active learning selection based on the loss prediction performs better than random baseline, entropy based acquisition and\ncore-set\n.\nActive learning results of loss prediction module based selection, in comparison with other approaches. (Image source:\nYoo & Kweon 2019\n)\nAdversarial Setup\n#\nSinha et al. (2019)\nproposed a GAN-like setup, named\nVAAL\n(Variational Adversarial Active Learning), where a discriminator is trained to distinguish unlabeled data from labeled data. Interestingly, active learning acquisition criteria does not depend on the task performance in VAAL.\nIllustration of VAAL (Variational adversarial active learning). (Image source:\nSinha et al. 2019\n)\nThe $\\beta$-VAE learns a latent feature space $\\mathbf{z}^l \\cup \\mathbf{z}^u$, for labeled and unlabeled data respectively, aiming to\ntrick\nthe discriminator $D(.)$ that all the data points are from the labeled pool;\nThe discriminator $D(.)$ predicts whether a sample is labeled (1) or not (0) based on a latent representation $\\mathbf{z}$. VAAL selects unlabeled samples with low discriminator scores, which indicates that those samples are sufficiently different from previously labeled ones.\nThe loss for VAE representation learning in VAAL contains both a reconstruction part (minimizing the ELBO of given samples) and an adversarial part (labeled and unlabeled data is drawn from the same probability distribution $q_\\phi$):\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{VAE} &= \\lambda_1 \\mathcal{L}^\\text{rec}_\\text{VAE} + \\lambda_2 \\mathcal{L}^\\text{adv}_\\text{VAE} \\\\\n\\mathcal{L}^\\text{rec}_\\text{VAE} &= \\mathbb{E}[\\log p_\\theta(\\mathbf{x}^l \\vert \\mathbf{z}^l)] - \\beta \\text{KL}(q_\\phi(\\mathbf{z}^l \\vert \\mathbf{x}^l) \\| p(\\mathbf{\\tilde{z}})) + \\mathbb{E}[\\log p_\\theta(\\mathbf{u} \\vert \\mathbf{z}^u)] - \\beta \\text{KL}(q_\\phi(\\mathbf{z}^u \\vert \\mathbf{u}) \\| p(\\mathbf{\\tilde{z}})) \\\\\n\\mathcal{L}^\\text{adv}_\\text{VAE} &= - \\mathbb{E}[\\log D(q_\\phi (\\mathbf{z}^l \\vert \\mathbf{x}^l))] - \\mathbb{E}[\\log D(q_\\phi(\\mathbf{z}^u \\vert \\mathbf{u}))]\n\\end{aligned}\n$$\nwhere $p(\\mathbf{\\tilde{z}})$ is a unit Gaussian as a predefined prior and $\\beta$ is the Lagrangian parameter.\nThe discriminator loss is:\n$$\n\\mathcal{L}_D = -\\mathbb{E}[\\log D(q_\\phi (\\mathbf{z}^l \\vert \\mathbf{x}^l))] - \\mathbb{E}[\\log (1 - D(q_\\phi (\\mathbf{z}^u \\vert \\mathbf{u})))]\n$$\nExperiment results of VAAL (variational adversarial active learning) on several image classification tasks. (Image source:\nSinha et al. 2019\nAblation studies showed that jointly training VAE and discriminator is critical. Their results are robust to the biased initial labeled pool, different labeling budgets and noisy oracle.\nMAL\n(Minimax Active Learning;\nEbrahimiet al. 2021\n) is an extension of VAAL. The MAL framework consists of an entropy minimizing feature encoding network $F$ followed by an entropy maximizing classifier $C$. This minimax setup reduces the distribution gap between labeled and unlabeled data.\nIllustration of the MAL (minimax active learning) framework. (Image source:\nEbrahimiet al. 2021\n)\nA feature encoder $F$ encodes a sample into a $\\ell_2$-normalized $d$-dimensional latent vector. Assuming there are $K$ classes, a classifier $C$ is parameterized by $\\mathbf{W} \\in \\mathbb{R}^{d \\times K}$.\n(1) First $F$ and $C$ are trained on labeled samples by a simple cross entropy loss to achieve good classification results,\n$$\n\\mathcal{L}_\\text{CE} = -\\mathbb{E}_{(\\mathbf{x}^l, y) \\sim \\mathcal{X}} \\sum_{k=1}^K \\mathbb{1}[k=y] \\log\\Big( \\sigma(\\frac{1}{T} \\frac{\\mathbf{W}^\\top F\\big(\\mathbf{x}^l)}{\\|F(\\mathbf{x}^l)\\|}\\big) \\Big)\n$$\n(2) When training on the unlabeled examples, MAL relies on a\nminimax\ngame setup\n$$\n\\begin{aligned}\n\\mathcal{L}_\\text{Ent} &= -\\sum^K_{k=1} p(y=k \\vert \\mathbf{u}) \\log p(y=k\\vert \\mathbf{u}) \\\\\n\\theta^*_F, \\theta^*_C &= \\min_F\\max_C \\mathcal{L}_\\text{Ent} \\\\\n\\theta_F &\\gets \\theta_F - \\alpha_1 \\nabla \\mathcal{L}_\\text{Ent} \\\\\n\\theta_C &\\gets \\theta_C + \\alpha_2 \\nabla \\mathcal{L}_\\text{Ent}\n\\end{aligned}\n$$\nwhere,\nFirst, minimizing the entropy in $F$ encourages unlabeled samples associated with similar predicted labels to have similar features.\nMaximizing the entropy in $C$ adversarially makes the prediction to follow a more uniform class distribution.\n(My understanding here is that because the true label of an unlabeled sample is unknown, we should not optimize the classifier to maximize the predicted labels just yet.)\nThe discriminator is trained in the same way as in VAAL.\nSampling strategy in MAL considers both diversity and uncertainty:\nDiversity: the score of $D$ indicates how similar a sample is to previously seen examples. A score closer to 0 is better to select unfamiliar data points.\nUncertainty: use the entropy obtained by $C$. A higher entropy score indicates that the model cannot make a confident prediction yet.\nThe experiments compared MAL to random, entropy, core-set, BALD and VAAL baselines, on image classification and segmentation tasks. The results look pretty strong.\nPerformance of MAL on ImageNet. (Table source:\nEbrahimiet al. 2021\n)\nCAL\n(Contrastive Active Learning;\nMargatina et al. 2021\n) intends to select\ncontrastive\nexamples. If two data points with different labels share similar network representations $\\Phi(.)$, they are considered as contrastive examples in CAL. Given a pair of contrastive examples $(\\mathbf{x}_i, \\mathbf{x}_j)$, they should\n$$\nd(\\Phi(\\mathbf{x}_i), \\Phi(\\mathbf{x}_j)) < \\epsilon \\quad\\text{and}\\quad \\text{KL}(p(y\\vert \\mathbf{x}_i) \\| p(y\\vert \\mathbf{x}_j)) \\rightarrow \\infty\n$$\nGiven an unlabeled sample $\\mathbf{x}$, CAL runs the following process:\nSelect the top $k$ nearest neighbors in the model feature space among the labeled samples, $\\{(\\mathbf{x}^l_i, y_i\\}_{i=1}^M \\subset \\mathcal{X}$.\nCompute the KL divergence between the model output probabilities of $\\mathbf{x}$ and each in $\\{\\mathbf{x}^l\\}$. The contrastive score of $\\mathbf{x}$ is the average of these KL divergence values: $s(\\mathbf{x}) = \\frac{1}{M} \\sum_{i=1}^M \\text{KL}(p(y \\vert \\mathbf{x}^l_i | p(y \\vert \\mathbf{x}))$.\nSamples with\nhigh contrastive scores\nare selected for active learning.\nOn a variety of classification tasks, the experiment results of CAL look similar to the entropy baseline.\nMeasuring Representativeness\n#\nCore-sets Approach\n#\nA\ncore-set\nis a concept in computational geometry, referring to a small set of points that approximates the shape of a larger point set. Approximation can be captured by some geometric measure. In the active learning, we expect a model that is trained over the core-set to behave comparably with the model on the entire data points.\nSener & Savarese (2018)\ntreats active learning as a core-set selection problem. Let’s say, there are $N$ samples in total accessible during training. During active learning, a small set of data points get labeled at every time step $t$, denoted as $\\mathcal{S}^{(t)}$. The upper bound of the learning objective can be written as follows, where the\ncore-set loss\nis defined as the difference between average empirical loss over the labeled samples and the loss over the entire dataset including unlabelled ones.\n$$\n\\begin{aligned}\n\\mathbb{E}_{(\\mathbf{x}, y) \\sim p} [\\mathcal{L}(\\mathbf{x}, y)]\n\\leq& \\bigg\\vert \\mathbb{E}_{(\\mathbf{x}, y) \\sim p} [\\mathcal{L}(\\mathbf{x}, y)] - \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}(\\mathbf{x}_i, y_i) \\bigg\\vert & \\text{; Generalization error}\\\\\n+& \\frac{1}{\\vert \\mathcal{S}^{(t)} \\vert} \\sum_{j=1}^{\\vert \\mathcal{S}^{(t)} \\vert} \\mathcal{L}(\\mathbf{x}^l_j, y_j) & \\text{; Training error}\\\\\n+& \\bigg\\vert \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}(\\mathbf{x}_i, y_i) - \\frac{1}{\\vert \\mathcal{S}^{(t)} \\vert} \\sum_{j=1}^{\\vert \\mathcal{S}^{(t)} \\vert} \\mathcal{L}(\\mathbf{x}^l_j, y_j) \\bigg\\vert & \\text{; Core-set error}\n\\end{aligned}\n$$\nThen the active learning problem can be redefined as:\n$$\n\\min_{\\mathcal{S}^{(t+1)} : \\vert \\mathcal{S}^{(t+1)} \\vert \\leq b} \\bigg\\vert \\frac{1}{N}\\sum_{i=1}^N \\mathcal{L}(\\mathbf{x}_i, y_i) - \\frac{1}{\\vert \\mathcal{S}^{(t)} \\cup \\mathcal{S}^{(t+1)} \\vert} \\sum_{j=1}^{\\vert \\mathcal{S}^{(t)} \\cup \\mathcal{S}^{(t+1)} \\vert} \\mathcal{L}(\\mathbf{x}^l_j, y_j) \\bigg\\vert\n$$\nIt is equivalent to\nthe $k$-Center problem\n: choose $b$ center points such that the largest distance between a data point and its nearest center is minimized. This problem is NP-hard. An approximate solution depends on the greedy algorithm.\nActive learning results of core-sets algorithm in comparison with several common baselines on CIFAR-10, CIFAR-100, SVHN. (Image source:\nSener & Savarese 2018\n)\nIt works well on image classification tasks when there is a small number of classes. When the number of classes grows to be large or the data dimensionality increases (“curse of dimensionality”), the core-set method becomes less effective (\nSinha et al. 2019\n).\nBecause the core-set selection is expensive,\nColeman et al. (2020)\nexperimented with a weaker model (e.g. smaller, weaker architecture, not fully trained) and found that empirically using a weaker model as a proxy can significantly shorten each repeated data selection cycle of training models and selecting samples, without hurting the final error much. Their method is referred to as\nSVP\n(Selection via Proxy).\nDiverse Gradient Embedding\n#\nBADGE\n(Batch Active learning by Diverse Gradient Embeddings;\nAsh et al. 2020\n) tracks both model uncertainty and data diversity in the gradient space. Uncertainty is measured by the gradient magnitude w.r.t. the final layer of the network and diversity is captured by a diverse set of samples that span in the gradient space.\nUncertainty. Given an unlabeled sample $\\mathbf{x}$, BADGE first computes the prediction $\\hat{y}$ and the gradient $g_\\mathbf{x}$ of the loss on $(\\mathbf{x}, \\hat{y})$ w.r.t. the last layer’s parameters. They observed that the norm of $g_\\mathbf{x}$ conservatively estimates the example’s influence on the model learning and high-confidence samples tend to have gradient embeddings of small magnitude.\nDiversity. Given many gradient embeddings of many samples, $g_\\mathbf{x}$, BADGE runs\n$k$-means++\nto sample data points accordingly.\nAlgorithm of BADGE (batch active learning by diverse gradient embeddings). (Image source:\nAsh et al. 2020\n)\nMeasuring Training Effects\n#\nQuantify Model Changes\n#\nSettles et al. (2008)\nintroduced an active learning query strategy, named\nEGL\n(Expected Gradient Length). The motivation is to find samples that can trigger the greatest update on the model if their labels are known.\nLet $\\nabla \\mathcal{L}(\\theta)$ be the gradient of the loss function with respect to the model parameters. Specifically, given an unlabeled sample $\\mathbf{x}_i$, we need to calculate the gradient assuming the label is $y \\in \\mathcal{Y}$, $\\nabla \\mathcal{L}^{(y)}(\\theta)$. Because the true label $y_i$ is unknown, EGL relies on the current model belief to compute the expected gradient change:\n$$\n\\text{EGL}(\\mathbf{x}_i) = \\sum_{y_i \\in \\mathcal{Y}} p(y=y_i \\vert \\mathbf{x}) \\|\\nabla \\mathcal{L}^{(y_i)}(\\theta)\\|\n$$\nBALD\n(Bayesian Active Learning by Disagreement;\nHoulsby et al. 2011\n) aims to identify samples to maximize the information gain about the model weights, that is equivalent to maximize the decrease in expected posterior entropy.\n$$\n\\begin{aligned}\nI[\\boldsymbol{\\theta}, y \\vert x,\\mathcal{D}] \n&= H(\\boldsymbol{\\theta} \\vert \\mathcal{D}) - \\mathbb{E}_{y \\sim p(y \\vert \\boldsymbol{x}, \\mathcal{D})} \\big[ H(\\boldsymbol{\\theta} \\vert y, \\boldsymbol{x}, \\mathcal{D}) \\big] & \\text{; Decrease in expected posterior entropy}\\\\ \n&= H(y \\vert \\boldsymbol{x}, \\mathcal{D}) - \\mathbb{E}_{\\boldsymbol{\\theta} \\sim p(\\boldsymbol{\\theta} \\vert \\mathcal{D})} \\big[ H(y \\vert \\boldsymbol{x}, \\mathcal{\\theta}) \\big]\n\\end{aligned}\n$$\nThe underlying interpretation is to “seek $\\mathbf{x}$ for which the model is marginally most uncertain about $y$ (high $H(y \\vert \\mathbf{x}, \\mathcal{D})$), but for which individual settings of the parameters are confident (low $H(y \\vert \\mathbf{x}, \\boldsymbol{\\theta})$).” In other words, each individual posterior draw is confident but a collection of draws carry diverse opinions.\nBALD was originally proposed for an individual sample and\nKirsch et al. (2019)\nextended it to work in batch mode.\nForgetting Events\n#\nTo investigate whether neural networks have a tendency to\nforget\npreviously learned information,\nMariya Toneva et al. (2019)\ndesigned an experiment: They track the model prediction for each sample during the training process and count the transitions for each sample from being classified correctly to incorrectly or vice-versa. Then samples can be categorized accordingly,\nForgettable\n(redundant) samples: If the class label changes across training epochs.\nUnforgettable\nsamples: If the class label assignment is consistent across training epochs. Those samples are never forgotten once learned.\nThey found that there are a large number of unforgettable examples that are never forgotten once learnt. Examples with noisy labels or images with “uncommon” features (visually complicated to classify) are among the most forgotten examples. The experiments empirically validated that unforgettable examples can be safely removed without compromising model performance.\nIn the implementation, the forgetting event is only counted when a sample is included in the current training batch; that is, they compute forgetting across presentations of the same example in subsequent mini-batches. The number of forgetting events per sample is quite stable across different seeds and forgettable examples have a small tendency to be first-time learned later in the training. The forgetting events are also found to be transferable throughout the training period and between architectures.\nForgetting events can be used as a signal for active learning acquisition if we hypothesize a model changing predictions during training is an indicator of model uncertainty. However, ground truth is unknown for unlabeled samples.\nBengar et al. (2021)\nproposed a new metric called\nlabel dispersion\nfor such a purpose. Let’s see across the training time, $c^*$ is the most commonly predicted label for the input $\\mathbf{x}$ and the label dispersion measures the fraction of training steps when the model does not assign $c^**$ to this sample:\n$$\n\\text{Dispersion}(\\mathbf{x}) = 1 - \\frac{f_\\mathbf{x}}{T} \\text{ where }\nf_\\mathbf{x} = \\sum_{t=1}^T \\mathbb{1}[\\hat{y}_t = c^*], c^* = \\arg\\max_{c=1,\\dots,C}\\sum_{t=1}^T \\mathbb{1}[\\hat{y}_t = c]\n$$\nIn their implementation, dispersion is computed at every epoch. Label dispersion is low if the model consistently assigns the same label to the same sample but high if the prediction changes often. Label dispersion is correlated with network uncertainty, as shown in\nLabel dispersion is correlated with network uncertainty. On the x-axis, data points are sorted by label dispersion scores. The y-axis is the model prediction accuracy when the model trys to infer the labels for those samples. (Image source:\nBengar et al. 2021\n)\nHybrid\n#\nWhen running active learning in batch mode, it is important to control diversity within a batch.\nSuggestive Annotation\n(\nSA\n;\nYang et al. 2017\n) is a two-step hybrid strategy, aiming to select both high uncertainty & highly representative labeled samples. It uses uncertainty obtained from an ensemble of models trained on the labeled data and core-sets for choosing representative data samples.\nFirst, SA selects top $K$ images with high uncertainty scores to form a candidate pool $\\mathcal{S}_c \\subseteq \\mathcal{S}_U$. The uncertainty is measured as disagreement between multiple models training with bootstrapping.\nThe next step is to find a subset $\\mathcal{S}_a \\subseteq \\mathcal{S}_c$ with highest representativeness. The cosine similarity between feature vectors of two inputs approximates how similar they are. The representativeness of $\\mathcal{S}_a$ for $\\mathcal{S}_U$ reflects how well $\\mathcal{S}_a$ can represent all the samples in $\\mathcal{S}_u$, defined as:\n$$\nF(\\mathcal{S}_a, \\mathcal{S}_u) = \\sum_{\\mathbf{x}_j \\in \\mathcal{S}_u} f(\\mathcal{S}_a, \\mathbf{x}_j) = \\sum_{\\mathbf{x}_j \\in \\mathcal{S}_u} \\max_{\\mathbf{x}_i \\in \\mathcal{S}_a} \\text{sim}(\\mathbf{x}_i, \\mathbf{x}_j)\n$$\nFormulating $\\mathcal{S}_a \\subseteq \\mathcal{S}_c$ with $k$ data points that maximizes $F(\\mathcal{S}_a, \\mathcal{S}_u)$ is a generalized version of the maximum set cover problem. It is NP-hard and its best possible polynomial time approximation algorithm is a simple greedy method.\nInitially, $\\mathcal{S}_a = \\emptyset$ and $F(\\mathcal{S}_a, \\mathcal{S}_u) = 0$.\nThen,  iteratively add $\\mathbf{x}_i \\in \\mathcal{S}_c$ that maximizes $F(\\mathcal{S}_a \\cup I_i, \\mathcal{S}_u)$ over $\\mathcal{S}_a$, until $\\mathcal{S}_s$ contains $k$ images.\nZhdanov (2019)\nruns a similar process as SA, but at step 2, it relies on $k$-means instead of core-set, where the size of the candidate pool is configured relative to the batch size. Given batch size $b$ and a constant $beta$ (between 10 and 50), it follows these steps:\nTrain a classifier on the labeled data;\nMeasure informativeness of every unlabeled example (e.g. using uncertainty metrics);\nPrefilter top $\\beta b \\geq b$ most informative examples;\nCluster $\\beta b$ examples into $B$ clusters;\nSelect $b$ different examples closest to the cluster centers for this round of active learning.\nActive learning can be further combined with\nsemi-supervised learning\nto save the budget.\nCEAL\n(Cost-Effective Active Learning;\nYang et al. 2017\n) runs two things in parallel:\nSelect uncertain samples via active learning and get them labeled;\nSelect samples with the most confident prediction and assign them\npseudo labels\n. The confidence prediction is judged by whether the prediction entropy is below a threshold $\\delta$. As the model is getting better in time, the threshold $\\delta$ decays in time as well.\nIllustration of CEAL (cost-effective active learning). (Image source:\nYang et al. 2017\n)\nCitation\n#\nCited as:\nWeng, Lilian. (Feb 2022). Learning with not enough data part 2: active learning. Lil’Log. https://lilianweng.github.io/posts/2022-02-20-active-learning/.\nOr\n@article{weng2022active,\n  title   = \"Learning with not Enough Data Part 2: Active Learning\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2022\",\n  month   = \"Feb\",\n  url     = \"https://lilianweng.github.io/posts/2022-02-20-active-learning/\"\n}\nReferences\n#\n[1] Burr Settles.\nActive learning literature survey.\nUniversity of Wisconsin, Madison, 52(55-66):11, 2010.\n[2]\nhttps://jacobgil.github.io/deeplearning/activelearning\n[3] Yang et al.\n“Cost-effective active learning for deep image classification”\nTCSVT 2016.\n[4] Yarin Gal et al.\n“Dropout as a Bayesian Approximation: representing model uncertainty in deep learning.”\nICML 2016.\n[5] Blundell et al.\n“Weight uncertainty in neural networks (Bayes-by-Backprop)”\nICML 2015.\n[6] Settles et al.\n“Multiple-Instance Active Learning.”\nNIPS 2007.\n[7] Houlsby et al.\nBayesian Active Learning for Classification and Preference Learning.\"\narXiv preprint arXiv:1112.5745 (2020).\n[8] Kirsch et al.\n“BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning.”\nNeurIPS 2019.\n[9] Beluch et al.\n“The power of ensembles for active learning in image classification.”\nCVPR 2018.\n[10] Sener & Savarese.\n“Active learning for convolutional neural networks: A core-set approach.”\nICLR 2018.\n[11] Donggeun Yoo & In So Kweon.\n“Learning Loss for Active Learning.”\nCVPR 2019.\n[12] Margatina et al.\n“Active Learning by Acquiring Contrastive Examples.”\nEMNLP 2021.\n[13] Sinha et al.\n“Variational Adversarial Active Learning”\nICCV 2019\n[14] Ebrahimiet al.\n“Minmax Active Learning”\narXiv preprint arXiv:2012.10467 (2021).\n[15] Mariya Toneva et al.\n“An empirical study of example forgetting during deep neural network learning.”\nICLR 2019.\n[16] Javad Zolfaghari Bengar et al.\n“When Deep Learners Change Their Mind: Learning Dynamics for Active Learning.”\nCAIP 2021.\n[17] Yang et al.\n“Suggestive annotation: A deep active learning framework for biomedical image segmentation.”\nMICCAI 2017.\n[18] Fedor Zhdanov.\n“Diverse mini-batch Active Learning”\narXiv preprint arXiv:1901.05954 (2019).",
  "metadata": {
    "title": "Learning with not Enough Data Part 2: Active Learning | Lil'Log",
    "description": "This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.",
    "keywords": "data, active-learning",
    "author": "Lilian Weng",
    "og:title": "Learning with not Enough Data Part 2: Active Learning",
    "og:description": "This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/",
    "twitter:card": "summary",
    "twitter:title": "Learning with not Enough Data Part 2: Active Learning",
    "twitter:description": "This is part 2 of what to do when facing a limited amount of labeled data for supervised learning tasks. This time we will get some amount of human labeling work involved, but within a budget limit, and therefore we need to be smart when selecting which samples to label.",
    "article:section": "posts",
    "article:published_time": "2022-02-20T00:00:00+00:00",
    "article:modified_time": "2022-02-20T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/028a8389cdec/code/body-000.txt",
      "chars": 276,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-workflow.png",
      "bytes": 251848,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-d81486b940074ca7bee2.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-workflow.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/DBAL-exp.png",
      "bytes": 255226,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-a6d3abf0912005132083.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/DBAL-exp.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-loss-prediction.png",
      "bytes": 208573,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-533dde5b3386ec869780.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-loss-prediction.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-loss-prediction-exp.png",
      "bytes": 582217,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-a27ab134b2539e4fa927.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/active-learning-loss-prediction-exp.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/VAAL.png",
      "bytes": 545888,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-f2a382c35839b44510af.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/VAAL.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/VAAL-exp.png",
      "bytes": 705319,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-5b0a42592fcac6b2269f.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/VAAL-exp.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/MAL.png",
      "bytes": 412179,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-f70d56c397cd0359db6b.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/MAL.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/MAL-exp.png",
      "bytes": 264815,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-e108c7cf704e84720825.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/MAL-exp.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/core-sets-exp.png",
      "bytes": 1047500,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-bd8c66b6e5605f2b18ac.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/core-sets-exp.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/BADGE-algo.png",
      "bytes": 246545,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-57159f6aa6eb14ac2a40.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/BADGE-algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/label-dispersion-vs-uncertainty.png",
      "bytes": 217492,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-cad8d44f6e9a7e9a6aef.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/label-dispersion-vs-uncertainty.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/CEAL.png",
      "bytes": 896102,
      "reused": false,
      "local_path": "attachments/028a8389cdec/images/body-5b32e280369a4496da10.png",
      "src": "https://lilianweng.github.io/posts/2022-02-20-active-learning/CEAL.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://burrsettles.com/pub/settles.activelearning.pdf",
      "label": "survey paper",
      "type": "a"
    },
    {
      "href": "https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.455.9057&rep=rep1&type=pdf",
      "label": "Der Kiureghian & Ditlevsen 2009",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1703.04977",
      "label": "Kendall & Gal 2017",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/AdaBoost",
      "label": "AdaBoost",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Bootstrapping_(statistics)",
      "label": "Bootstrapping",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Gradient_boosting",
      "label": "GBM",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1602.04621",
      "label": "Osband, et al. 2016",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2020-06-07-exploration-drl/",
      "label": "exploration",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1506.02142",
      "label": "Gal & Ghahramani 2016",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1506.02157",
      "label": "Gal & Ghahramani 2016",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1703.02910",
      "label": "Gal et al. 2017",
      "type": "a"
    },
    {
      "href": "https://openaccess.thecvf.com/content_cvpr_2018/papers/Beluch_The_Power_of_CVPR_2018_paper.pdf",
      "label": "Beluch et al. (2018)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1811.03897",
      "label": "Pop & Fulop 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1505.05424",
      "label": "Blundell et al. 2015",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.03677",
      "label": "Yoo & Kweon (2019)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.00370",
      "label": "Sinha et al. (2019)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2012.10467",
      "label": "Ebrahimiet al. 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2109.03764",
      "label": "Margatina et al. 2021",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2021-05-31-contrastive/",
      "label": "contrastive",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1708.00489",
      "label": "Sener & Savarese (2018)",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Metric_k-center",
      "label": "the $k$-Center problem",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1906.11829",
      "label": "Coleman et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1906.03671",
      "label": "Ash et al. 2020",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/K-means%2B%2B",
      "label": "$k$-means++",
      "type": "a"
    },
    {
      "href": "https://papers.nips.cc/paper/2007/hash/a1519de5b5d44b31a01de013b9b51a80-Abstract.html",
      "label": "Settles et al. (2008)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1112.5745",
      "label": "Houlsby et al. 2011",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1906.08158",
      "label": "Kirsch et al. (2019)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1812.05159",
      "label": "Mariya Toneva et al. (2019)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2107.14707",
      "label": "Bengar et al. (2021)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.04737",
      "label": "Yang et al. 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1901.05954",
      "label": "Zhdanov (2019)",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2021-12-05-semi-supervised/",
      "label": "semi-supervised learning",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1701.03551",
      "label": "Yang et al. 2017",
      "type": "a"
    },
    {
      "href": "https://jacobgil.github.io/deeplearning/activelearning",
      "label": "https://jacobgil.github.io/deeplearning/activelearning",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:24Z",
  "source_url": "https://lilianweng.github.io/posts/2022-02-20-active-learning/",
  "papers": [
    {
      "status": "ok",
      "url": "https://burrsettles.com/pub/settles.activelearning.pdf",
      "bytes": 1939936,
      "reused": false,
      "local_path": "papers/repair/1a9a9c6742fab449158e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.04977",
      "bytes": 6719323,
      "reused": false,
      "local_path": "papers/repair/266ae612868b87b732a4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1602.04621",
      "bytes": 6874097,
      "reused": false,
      "local_path": "papers/repair/d7a1d129f52a82d595ff.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1506.02142",
      "bytes": 1107246,
      "reused": false,
      "local_path": "papers/repair/4fb011fb3475ef624220.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1506.02157",
      "bytes": 436231,
      "reused": false,
      "local_path": "papers/repair/f341fa9dd0f0f593ee60.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.02910",
      "bytes": 724240,
      "reused": false,
      "local_path": "papers/repair/523334b8d44dca5de9f0.pdf"
    },
    {
      "status": "ok",
      "url": "https://openaccess.thecvf.com/content_cvpr_2018/papers/Beluch_The_Power_of_CVPR_2018_paper.pdf",
      "bytes": 1715354,
      "reused": false,
      "local_path": "papers/repair/f0bcca0541114ac25b32.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1811.03897",
      "bytes": 4205547,
      "reused": false,
      "local_path": "papers/repair/ce97c4ced1411688db16.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1505.05424",
      "bytes": 628773,
      "reused": false,
      "local_path": "papers/repair/475ff08ad5512624a7cc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.03677",
      "bytes": 564131,
      "reused": false,
      "local_path": "papers/repair/79dc6ada83281404ff9d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.00370",
      "bytes": 930575,
      "reused": false,
      "local_path": "papers/repair/79c5aedc49e45da7dda8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2012.10467",
      "bytes": 1339851,
      "reused": false,
      "local_path": "papers/repair/90cd58e430cea1408452.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2109.03764",
      "bytes": 2036447,
      "reused": false,
      "local_path": "papers/repair/f5ba7910af8e3a2395ac.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1708.00489",
      "bytes": 5033708,
      "reused": false,
      "local_path": "papers/repair/238fc8d82668ad254fc1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1906.11829",
      "bytes": 17351339,
      "reused": false,
      "local_path": "papers/repair/ad0851b7cc881c207705.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1906.03671",
      "bytes": 4265486,
      "reused": false,
      "local_path": "papers/repair/7474699a4e844a465ace.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1112.5745",
      "bytes": 556354,
      "reused": false,
      "local_path": "papers/repair/d0182f4c21330f972a26.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1906.08158",
      "bytes": 4001227,
      "reused": false,
      "local_path": "papers/repair/b525dbbc20e226a0e95a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1812.05159",
      "bytes": 1731989,
      "reused": true,
      "local_path": "papers/repair/55cf5f36e50e938900cd.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2107.14707",
      "bytes": 488570,
      "reused": false,
      "local_path": "papers/repair/27448d46f2a1e682c257.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.04737",
      "bytes": 5679851,
      "reused": false,
      "local_path": "papers/repair/de814b520904eeb1ed6e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1901.05954",
      "bytes": 387045,
      "reused": false,
      "local_path": "papers/repair/3bd458965ad0d9c6d253.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1701.03551",
      "bytes": 2102878,
      "reused": false,
      "local_path": "papers/repair/85a5a6a9cc2ce15af84e.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 32971,
  "expected_body_images": 12,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/028a8389cdec/pages/source.html",
  "local_html": "attachments/028a8389cdec/pages/article.html",
  "body_base_path": "attachments/028a8389cdec/pages/",
  "body_markdown_path": "attachments/028a8389cdec/pages/article.md"
}