{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Eugene Yan",
  "title_listed": "Bootstrapping Labels via ___ Supervision & Human-In-The-Loop",
  "keywords": "弱监督,标注,人工",
  "domain": "数据处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://eugeneyan.com/writing/bootstrapping-data-labels/",
  "publish_date": "2021-10",
  "corpus_stage": "数据处理",
  "notes": "通过弱监督与人工标注引导标签",
  "id": "bf4a79d2f0e9",
  "collected_at": "2026-10-02T14:16:59",
  "title": "Bootstrapping Labels via ___ Supervision & Human-In-The-Loop",
  "body_markdown": "eugeneyan\nStart Here\nWriting\nSpeaking\nPrototyping\nAbout\nBootstrapping Labels via ___ Supervision & Human-In-The-Loop\n[\nteardown\nmachinelearning\n]\n· 12 min read\nMost machine learning tutorials and papers assume the availability of training labels. This includes benchmark datasets such as\nOpenImages\nor\nSuperGLUE\n, or customer interaction behavior such as clicks or purchases.\nBut what if labeled datasets are not available?\nWe would have to collect them. Collecting training labels is a seldom discussed art. Imagine you’re monitoring social media for undesirable content—how would you get labels for subtle concepts like COVID-19 misinformation, terrorism, or\ncyberbullying\n? Or imagine you need to tag a corpus of book reviews to indicate if the review is about quality, pricing, or writing style—where would you begin?\nIn this write-up, we’ll discuss semi, active, and weakly supervised learning, and see examples of the latter two approaches from DoorDash, Facebook, Google, and Apple.\nSemi-supervised learning: Generate pseudo-labels\nIn semi-supervised learning\n, we combine a small amount of hand-labeled data with a larger amount of unlabeled data during training. Here’s a step-by-step:\nTrain a high-precision model on labeled data\nPredict on unlabeled data\nSelect the most confident predictions as pseudo-labels; add them to training data\nTrain another model on labels and pseudo-labels\nRepeat until you have sufficient high confidence pseudo-labels\nI haven’t come across any papers sharing about semi-supervised learning applied in industry. Nonetheless, it’s frequently used in Kaggle competitions (e.g., pseudo-labels from the test dataset added as training data). Here’s an example of a\n4th place submission\nusing semi-supervised learning on audio tagging. (Also see this Google\npaper\non how adversaries can “poison” unlabeled samples and cause models trained via semi-supervised labels to misclassify.\nActive learning: Find samples for human-in-the-loop\nIn active learning\n, we select the most “interesting” unlabeled samples for labeling via human-in-the-loop (HITL). One way to identify such samples is uncertainty sampling, where samples with the highest uncertainty in model predictions are selected. For example, if the model is a binary classifier, uncertainty sampling selects samples where the predicted probability is close to 0.5. Query-by-committee extends uncertainty sampling with multiple models—samples that models disagree on are considered the most informative and selected.\nAnother approach is\ninformation density\n, where points that are most uncertain\nand\nrepresentative of the data distribution are selected. Assume we have two samples that have similar uncertainty. One has many samples close to it, while the other is an outlier. In this case, we would pick the sample that has other samples close to it.\nWhile point A has highest uncertainty, B has a mix of uncertainty and information density (\nsource\n)\nDoordash shared how they used active learning and HITL\nto\novercome the cold-start problem for menu item tagging\n. First, they\ncreated some seed data\nusing a simple, high-precision classifier to select samples for annotation—samples were selected with a higher probability threshold. They also shared about using unsupervised semantic embeddings to identify ambiguous samples for annotation. Nonetheless, it’s unclear how they trained the classifier (without labeled data), or how the semantic embeddings were used to select annotation samples—if you have more details, please reach out.\nDoordash's active learning workflow, with automated steps in green and HITL steps in red (\nsource\n)\nData augmentation\nwas then applied to the seed data by identifying unlabeled samples close to the labeled samples (in terms of edit distance and embedding cosine similarity) and added as training data. They also used random text augmentation by varying sentence order in item description and randomly remove information (e.g., menu categories). A ratio of 100 synthetic samples to one labeled sample was used for training.\nSingle-layer, multi-class LSTMs with FastText word embeddings were trained for each attribute group\n(e.g., regional cuisine style, flavor). Inputs included item names, menu categories, and full-text descriptions. They also tried multi-task models but found that they underperformed as sampling didn’t preserve actual tag distribution and there were too few samples to only train on labeled data. Nonetheless, as more labeled data was collected, the performance of multi-task models improved.\nTo select more samples for annotation\n, they focused on precision and recall. To improve precision, they selected samples similar to those where the model prediction conflicted with the annotator label. To improve recall, they used the model to predict on the unlabeled dataset and select samples that the model had low confidence on. The selected samples were sent to Mechanical Turk for a first pass; the more ambiguous samples were then sent to professional annotators for a second pass.\nThe primary metric used in\nlabel verification\nwas cross-rater agreement. Nonetheless, they also measured annotation agreement between different vendors to assess for systematic bias. In addition, they used annotated samples to build a small set of “golden” data which had high-confidence labels. Then, when onboarding new annotators, they mixed in the golden data to check if the baseline precision met their requirements.\nFacebook added a twist to active learning by including similarity search as a filter\n(\nSimilarity Search for Efficient Active Learning and Search\naka SEALS). They shared that, data is often heavily skewed, with only a small fraction of the data being relevant (ie., 1 in 1,000 or more). Thus, they use nearest neighbors on positive labels before active learning. Using nearest neighbors as a filter reduces the number of unlabeled samples to consider in the active learning step and increases annotation efficiency.\nThey tested this with ImageNet and OpenImages datasets. On ImageNet, they showed that SEALs performed similarly to baseline approaches that had full supervision (i.e., full labeled data) with only 2,000 labels per concept.—this was only 5% of the total samples in the candidate annotation pool. On OpenImages, they were able to match active learning approaches while only considering a fraction of the data.\nNearest neighbors allowed similar performance (as active learning) with 2 - 15% of data (\nsource\n)\nWeak supervision: Get lots of lower-quality labels fast\nWeak supervision blends knowledge from multiple data sources to quickly get lots of labels efficiently, albeit at a lower quality.\nSources include labeling functions that formalize heuristics via regex or aggregated statistics (i.e., data programming), or existing resources such as knowledge bases, alternative datasets, or pre-trained models. While labels from weak supervision may not be as accurate as hand-labeled data, they’re good enough to obtain decent performance.\nGoogle shared how they applied\nweak supervision via Snorkel DryBell\n.\nIt uses labeling functions that take in data points and either return a label or abstain. Labeling functions can be used to operationalize heuristics about the data source or content. They can also be based on internal models (e.g., internally developed topic models, named-entity recognizers) or graphs (e.g., user knowledge or entity graphs to derive labels). Here’s an example labeling function template.\nTemplate labeling function written in C++ (\nsource\n)\nThen, a generative model is used to estimate the accuracies of the various functions based on their observed agreements and disagreements. These accuracies are then used to re-weight and combine labels across labeling functions.\nHere’s an overview of Snorkel DryBell. Developers write labeling functions based on heuristics, existing models (e.g., internal NLP, topic models), or organization resources (e.g., knowledge graphs). Snorkel DryBell then applies these labeling functions on unlabeled examples before loading the output and labeling functions into a generative model. The model then combines them and returns probabilistic labels for use in production systems.\nAn overview of Snorkel Drybell outline in dashed blue lines (\nsource\n)\nIt was adopted in two use cases, one for topic classification and one for product classification. For topic classification, a developer wrote 10 labeling functions to apply heuristics based on the linked URL (URL-based), custom named-entity recognition models (NER-tagger-based), and internal topic models (topic model-based). For product classification, they used eight labeling functions based on keywords in the content (keyword-based), translations of keywords in 10 languages via the knowledge graph (graph-based), and semantic models to identify content unrelated to product categories of interest (model-based).\nIn both cases, Snorkel DryBell had higher predicted accuracy than data trained directly on a small, hand-labeled development set (which was also used to formulate labeling functions, hyperparameter tuning, etc.) To match the performance of Snorkel DryBell, it required 80k hand-labeled samples for topic classification and 12k hand-labeled samples for product classification.\nApple also shared a similar system (\nOverton\n).\nWhen new features are created or quality bugs are identified, engineers can provide additional supervision via data programming or other heuristics. Lineage for each source of information is also tracked. They also shared that there are production systems with no traditional supervised training data, which is important for privacy-conscious applications.\nThere are also other examples of weak supervision in the wild, such as IBM’s\nbootstrapping of conversational agents\nand Intel’s\nweak supervision on imbalanced extraction problems\n. Facebook also shared about\ndialogue learning with human-in-the-loop\n.\nAlso critical: Labeling guidelines & creative thinking\nAn under-appreciated aspect of gathering labeled data is defining labeling guidelines. While tasks such as labeling porn or hate speech might be relatively straightforward, how would you define labeling guidelines for COVID-19 misinformation? It might blend truth about COVID-19 with misinformation about vaccines, or could be nuanced with sarcasm. Even Tesla, which probably has one of the most sophisticated data labeling systems to support Autopilot, hasn’t worked out all the kinks.\nEven after 4 years I still haven't \"solved\" labeling workflows. Labeling, QA, Final QA, auto-labeling, error-spotting, diversity massaging, labeling docs + versioning, ppl training, escalations, data cleaning, throughput & quality stats, eval sets + categorization & boosting, ...\n— Andrej Karpathy (@karpathy)\nJuly 8, 2021\nDoorDash shared some tips to develop good labeling guidelines for tags, such as:\nEnsure tags are mutually exclusive (i.e., no overlap): This allows annotators to move on if a sample already has a tag and reduces annotation volume.\nPartition taxonomy at the top level by distinct attributes (e.g., regional cuisine style, flavor, etc.): This allows parallelization of annotation tasks.\nEnsure there’s a tag for “Other” at each level: This allows you to revisit the “Other” tags to improve granularity in future (instead of having to go through all tags).\nEnsure tags are objective: Tags such as “popular” or “convenient” are  subjective and can change over time.\nIt also helps to think beyond HITL labeling or semi/active/weak supervision. Try searching for labels creatively in the user data. If we’re trying to tag books with various attributes based on their content (e.g., Russia, Fae, etc), examine\nhow users organize their bookshelves\non GoodReads. These bookshelf names can be used as weak labels to seed tags.\nAlternatively, we can start with approaches that don’t require user data. How do we build semantic search without user interaction labels? One way is to start with\nlexical approaches\nsuch as\nBM25\nor whatever’s available in ElasticSearch or Solr. Once deployed, we can then collect user interaction data (e.g., clicks, purchases) on search results, which can then be used as labels for semantic search models.\nOther approaches to consider include crowdsourcing from users (e.g., thumbs up/down on recommendations or search results), synthetic data generation (e.g.,\nVAEs\n,\nGANs\n), label propagation via graphs, or transfer learning using models trained on other tasks.\nHow to bootstrap labels at various stages of maturity\nGetting labels for machine learning systems seems more art than science. There’s no straightforward step-by-step and it’ll vary based on intended use (e.g., customer-facing labels have higher requirements than internal labels) and data type (e.g., potential fraud events, chatbot dialogue, self-driving video). Nonetheless, here’s a rough heuristic on when to apply which technique depending on the maturity of your ML systems.\nIn the earliest stage, we can adopt weak supervision. This can include labeling functions via keyword-based regex, statistical aggregations, knowledge graphs, user segmentation, etc. If necessary, we can apply HITL to verify the labels before using high-confidence labels to train models.\nWith some seed data, we can then adopt semi-supervised learning. We can train a high precision model to predict on unlabeled samples, then add the pseudo-labels with the highest confidence to the training data. Or we can try active learning (e.g., nearest neighbors + uncertainty sampling/information density) to identify labels with the highest uncertainty for HITL before adding it to the dataset.\nFinally, in the late stage, our machine learning systems and policies would have matured. Here’s where we start to get a stream of good labels for model training, and can try more sophisticated models that require a large number of labels. These models may eventually be strong enough for auto-decisioning, with the lowest confidence predictions being sent to HITL for confirmation.\nAlso consider the maturity of the labels. Even in mature ML systems, we might not be able to use the labels immediately as they can change over time. For example, the purchase that counted towards conversion might be returned within the 14-day return window, or that initially legitimate transaction might turn out to be fraudulent. Using premature or incorrect labels can lead to vicious cycle feedback loops, where the model learns from invalid labels and makes prediction errors that exacerbate ground truth errors.\nDid I miss any key techniques for bootstrapping labels? Please\nreach out\nand let me know!\nMany machine learning papers and tutorials start from labeled datasets. But what if we don't have labels? How do we collect them?\nHere we discuss semi-supervised learning, active learning, and weak supervision to bootstrap labels for your ML system.\nhttps://t.co/RvsgXCFlIK\n— Eugene Yan (@eugeneyan)\nAugust 4, 2021\nThanks to\nLim Wen Qing\nand\nNihit Desai\nfor pointing me to great resources. Also thanks to Nihit Desai and\nSamuel Flender\nfor providing feedback.\nIf you found this useful, please cite this write-up as:\nYan, Ziyou. (Aug 2021). Bootstrapping Labels via ___ Supervision & Human-In-The-Loop. eugeneyan.com.\n        https://eugeneyan.com/writing/bootstrapping-data-labels/.\nor\n@article{yan2021labeling,\n  title   = {Bootstrapping Labels via ___ Supervision & Human-In-The-Loop},\n  author  = {Yan, Ziyou},\n  journal = {eugeneyan.com},\n  year    = {2021},\n  month   = {Aug},\n  url     = {https://eugeneyan.com/writing/bootstrapping-data-labels/}\n}\nShare on:\nBrowse related tags:\n[\nteardown\nmachinelearning\n]\nor\nSearch\n« Mailbag: How to Bootstrap Labels for Relevant Docs in Search\nAmazon Science - Eugene Yan and the Art of Writing about Science »\nJoin\n11,800+\nreaders getting updates on machine learning, RecSys, LLMs, and engineering.\nGet email updates",
  "metadata": {
    "title": "Bootstrapping Labels via ___ Supervision & Human-In-The-Loop",
    "description": "How to generate labels from scratch with semi, active, and weakly supervised learning.",
    "keywords": null,
    "author": "Eugene Yan",
    "og:site_name": "eugeneyan.com",
    "og:title": "Bootstrapping Labels via ___ Supervision & Human-In-The-Loop",
    "og:type": "article",
    "og:description": "How to generate labels from scratch with semi, active, and weakly supervised learning.",
    "og:url": "https://eugeneyan.com/writing/bootstrapping-data-labels/",
    "og:image": "https://eugeneyan.com/assets/og_image/fruit-labels.jpg",
    "twitter:card": "summary_large_image",
    "twitter:domain": "eugeneyan.com",
    "twitter:title": "Bootstrapping Labels via ___ Supervision & Human-In-The-Loop",
    "twitter:description": "How to generate labels from scratch with semi, active, and weakly supervised learning.",
    "twitter:image": "https://eugeneyan.com/assets/og_image/fruit-labels.jpg",
    "article:published_time": "2021-08-01T00:00:00+00:00",
    "article:author": "https://eugeneyan.com/about/",
    "article:section": "posts",
    "article:tag": "machinelearning"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/bf4a79d2f0e9/code/body-000.txt",
      "chars": 268,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-search.svg",
      "bytes": 425,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-feeb4bd87b8704a903b2.svg",
      "src": "https://eugeneyan.com/assets/icon-search.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/information-density.webp",
      "bytes": 6038,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-9601862f90f4ca19d38c.webp",
      "src": "https://eugeneyan.com/assets/information-density.webp",
      "alt": "While point A has highest uncertainty, B has a mix of uncertainty and information density",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/doordash-active-learning.webp",
      "bytes": 11144,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-d6d3fffb4efd131090d7.webp",
      "src": "https://eugeneyan.com/assets/doordash-active-learning.webp",
      "alt": "Doordash's active learning workflow, with automated steps in green and HITL steps in red",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/facebook-seals.webp",
      "bytes": 47690,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-2f58f29d9ecc723beb65.webp",
      "src": "https://eugeneyan.com/assets/facebook-seals.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/snorkel-drybell-template.webp",
      "bytes": 22500,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-5b68d792d1694603098d.webp",
      "src": "https://eugeneyan.com/assets/snorkel-drybell-template.webp",
      "alt": "Template labeling function written in C++",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/snorkel-drybell-design.webp",
      "bytes": 26916,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-e99243aa35eec07c6124.webp",
      "src": "https://eugeneyan.com/assets/snorkel-drybell-design.webp",
      "alt": "Image",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-twitter.svg",
      "bytes": 892,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-0083a7581b07e5f40e7b.svg",
      "src": "https://eugeneyan.com/assets/icon-twitter.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "bytes": 559,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-f7a16c3e46e5838cec4d.svg",
      "src": "https://eugeneyan.com/assets/icon-linkedin.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/bluesky.svg",
      "bytes": 745,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-be754e8ecdaf6de94f55.svg",
      "src": "https://eugeneyan.com/assets/bluesky.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-facebook.svg",
      "bytes": 1123,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-5d55b59733923cf14223.svg",
      "src": "https://eugeneyan.com/assets/icon-facebook.svg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://eugeneyan.com/assets/icon-mail.svg",
      "bytes": 841,
      "reused": false,
      "local_path": "attachments/bf4a79d2f0e9/images/body-b179147617228dfafb9f.svg",
      "src": "https://eugeneyan.com/assets/icon-mail.svg",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=6q7c6Dlw0JI",
      "status": "ok",
      "local_path": "attachments/bf4a79d2f0e9/videos/6q7c6Dlw0JI.mp4",
      "bytes": 91671009,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "389.241000"
    }
  ],
  "links": [
    {
      "href": "https://eugeneyan.com/",
      "label": "eugeneyan",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/start-here/",
      "label": "Start Here",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/",
      "label": "Writing",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/speaking/",
      "label": "Speaking",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/prototyping/",
      "label": "Prototyping",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/about/",
      "label": "About",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/search/",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/teardown/",
      "label": "teardown",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/tag/machinelearning/",
      "label": "machinelearning",
      "type": "a"
    },
    {
      "href": "https://opensource.google/projects/open-images-dataset",
      "label": "OpenImages",
      "type": "a"
    },
    {
      "href": "https://super.gluebenchmark.com/",
      "label": "SuperGLUE",
      "type": "a"
    },
    {
      "href": "https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge",
      "label": "cyberbullying",
      "type": "a"
    },
    {
      "href": "https://www.kaggle.com/c/freesound-audio-tagging-2019/discussion/96440",
      "label": "4th place submission",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2105.01622",
      "label": "paper",
      "type": "a"
    },
    {
      "href": "https://dl.acm.org/doi/10.5555/1613715.1613855",
      "label": "information density",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=6q7c6Dlw0JI",
      "label": "source",
      "type": "a"
    },
    {
      "href": "https://doordash.engineering/2020/08/28/overcome-the-cold-start-problem-in-menu-item-tagging/",
      "label": "overcome the cold-start problem for menu item tagging",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2007.00077",
      "label": "Similarity Search for Efficient Active Learning and Search",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1812.00417",
      "label": "weak supervision via Snorkel DryBell",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1909.05372",
      "label": "Overton",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1812.06176",
      "label": "bootstrapping of conversational agents",
      "type": "a"
    },
    {
      "href": "https://dl.acm.org/doi/abs/10.1145/3329486.3329492",
      "label": "weak supervision on imbalanced extraction problems",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1611.09823",
      "label": "dialogue learning with human-in-the-loop",
      "type": "a"
    },
    {
      "href": "https://twitter.com/karpathy/status/1413242233394929664?ref_src=twsrc%5Etfw",
      "label": "July 8, 2021",
      "type": "a"
    },
    {
      "href": "https://www.goodreads.com/blog/show/1399-goodreads-hack-how-to-create-custom-bookshelves",
      "label": "how users organize their bookshelves",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/search-query-matching/",
      "label": "lexical approaches",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Okapi_BM25",
      "label": "BM25",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Variational_autoencoder",
      "label": "VAEs",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Generative_adversarial_network",
      "label": "GANs",
      "type": "a"
    },
    {
      "href": "https://twitter.com/eugeneyan",
      "label": "reach out",
      "type": "a"
    },
    {
      "href": "https://t.co/RvsgXCFlIK",
      "label": "https://t.co/RvsgXCFlIK",
      "type": "a"
    },
    {
      "href": "https://twitter.com/eugeneyan/status/1422713392914059265?ref_src=twsrc%5Etfw",
      "label": "August 4, 2021",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/wen-qing-lim-67330771/",
      "label": "Lim Wen Qing",
      "type": "a"
    },
    {
      "href": "https://twitter.com/nihit_desai",
      "label": "Nihit Desai",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/in/sflender/",
      "label": "Samuel Flender",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/writing/mailbag-bootstrap-relevant-docs/",
      "label": "« Mailbag: How to Bootstrap Labels for Relevant Docs in Search",
      "type": "a"
    },
    {
      "href": "https://eugeneyan.com/speaking/amazon-science-feature/",
      "label": "Amazon Science - Eugene Yan and the Art of Writing about Science »",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:20Z",
  "source_url": "https://eugeneyan.com/writing/bootstrapping-data-labels/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2105.01622",
      "bytes": 681816,
      "reused": false,
      "local_path": "papers/repair/fbeed10d9423a9752fbb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2007.00077",
      "bytes": 2031238,
      "reused": false,
      "local_path": "papers/repair/0c0d67290d4926dd41fd.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1812.00417",
      "bytes": 1187190,
      "reused": false,
      "local_path": "papers/repair/2cbdafe7ad1e3f57f160.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1909.05372",
      "bytes": 727275,
      "reused": false,
      "local_path": "papers/repair/cb26fa7d1eaf6bcaa38a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1812.06176",
      "bytes": 345222,
      "reused": false,
      "local_path": "papers/repair/5587506af481e7fe2c34.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1611.09823",
      "bytes": 1212274,
      "reused": false,
      "local_path": "papers/repair/cf1e36f16ff9777b5dcf.pdf"
    }
  ],
  "actual_video_files": [
    {
      "local_path": "attachments/bf4a79d2f0e9/videos/6q7c6Dlw0JI.mp4",
      "bytes": 91671009,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "389.241000",
      "status": "ok"
    }
  ],
  "body_chars": 15912,
  "expected_body_images": 11,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/bf4a79d2f0e9/pages/source.html",
  "local_html": "attachments/bf4a79d2f0e9/pages/article.html",
  "body_base_path": "attachments/bf4a79d2f0e9/pages/",
  "body_markdown_path": "attachments/bf4a79d2f0e9/pages/article.md"
}