{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "Reducing Toxicity in Language Models",
  "keywords": "毒性控制,安全,语言模型",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/",
  "publish_date": "2021-03",
  "corpus_stage": "强化学习/对齐",
  "notes": "语言模型毒性控制综述",
  "id": "5bdaf053b66c",
  "collected_at": "2026-10-02T14:13:10",
  "title": "Reducing Toxicity in Language Models | Lil'Log",
  "body_markdown": "Large pretrained\nlanguage models\nare trained over a sizable collection of online data. They unavoidably acquire certain toxic behavior and biases from the Internet. Pretrained language models are very powerful and have shown great success in many NLP tasks. However, to safely deploy them for practical real-world applications demands a strong safety control over the model generation process.\nMany challenges are associated with the effort to diminish various types of unsafe content:\nFirst, there are a variety of unsafe content types, such as toxicity, abusiveness, hate speech, biases, stereotypes, cyberbullying, identity attacks and more, which may or may not demand different treatment.\nSecond, there is no clearly and widely agreed-upon categorization and definition of unsafe behavior in pretrained language models. Individual perceptions could vary a lot due to different social backgrounds.\nIn this post, we delve into the issue of toxicity in language models. As I’m still struggling to find a concrete definition of toxic content, I list a couple in the literature below.\n[\nPerspective API\n] A rude, disrespectful, or unreasonable comment; likely to make people leave a discussion.\n[\nKurita et al. 2019\n] Content that can offend or harm its recipients, including hate speech, racism, and offensive language.\n[\nPavlopoulos et al. 2020\n] We use the term ’toxic’ as an umbrella term, but we note that the literature uses several terms for different kinds of toxic language or related phenomena: ‘offensive’, ‘abusive’, ‘hateful’, etc.\nOverall, toxicity is a broad term to describe several types of unsafe content. Methodologies in this post can be applied given some form of definition of toxicity; e.g. presented in the instruction for annotators. How to properly define the concept of toxicity and thus collect accurate annotation labels is out of the scope of this post.\nCategorization of Toxic Content\n#\nHow to categorize toxic content is not a straightforward task. Which content should be considered toxic and what types of toxic content exist can be very subjective. Language that does not look offensive to one group might seem inappropriate to another.\nOne popular categorization of offensive language is proposed by\nZampieri et al. (2019)\n, a three-level hierarchical taxonomy considering both the type and the target of offense. The Offensive Language Identification Dataset (\nOLID\n) dataset is collected based on this taxonomy.\nThe three-level hierarchical taxonomy for categorizing offensive language, proposed by\nZampieri et al. (2019)\n.\nLevel A: “Is it offensive?”\n[OFF]\nOffensive: Inappropriate language, insults, or threats.\n[NOT]\nNot offensive: No offense or profanity.\nLevel B: “Is the offensive text targeted?”\n[TIN]\nTargeted Insult: Targeted insult or threat towards an individual, a group or other.\n[UNT]\nUntargeted: Non-targeted profanity and swearing.\nLevel C: What is the target?\n[IND]\nThe offense targets an individual, often defined as “cyberbullying”.\n[GRP]\nThe offense targets a group of people based on ethnicity, gender, sexual orientation, religion, or other common characteristic, often defined as “hate speech”.\n[OTH]\nThe target can belong to other categories, such as an organization, an event, an issue, etc.\nData Collection\n#\nPreparing a dataset of samples labelled as “safe” vs “unsafe” is the foundation for training a toxic language classifier and further providing signals for model detoxification.\nHuman Annotations\n#\nVidgen & Derczynski (2020)\nsummarized that training data annotations for toxicity detection on the high level can be collected by:\nExpert coding\n: An expert has enough knowledge or training to complete the annotation tasks with good quality, such as a researcher who studies prejudice, a student with moderate level of training, or a NLP practitioner. It is more expensive but produces high-quality data.\nCrowdsourcing\n: Crowdsourcing platform pairs a large number of non-expert annotators with tasks. It is easier to scale up but demands more attention on quality control.\nProfessional moderators\n: Professional moderators are experienced, well-trained on the tasks, but their goals are likely to optimize for the output specific to the platform.\nSynthetic data\n: Training dataset can also be manually created by relevant content creators to cover a broad range of toxic content types.\nCrowdsourcing is the most common approach among them (\nDavidson et al. 2017\n,\nZampieri et al. 2019\n) and there are several good practices to improve the data quality:\nTest data\n:  A small set of annotations collected from a few experts can be used as test questions (\nZampieri et al. 2019\n) to filter out human annotators on the crowdsourcing platform who cannot achieve a certain threshold.\nClear guidelines\n: Detailed instructions are useful to guide annotators to produce aligned and consistent labels. Without any guideline, annotators are encouraged to apply their personal perceptions, which could be problematic because (1) subjective interpretation of toxic content varies across individuals greatly and (2) it is tricky to mark certain types of noise like sarcasm and irony without any guideline.\nMajority vote\n: It is very common that we need labels from multiple annotators per sample and take the majority vote.\nUnderstanding annotators’ identities\n: Demographic background has a big impact on the annotator’s understanding of the task. We should aim to recruit diverse and qualified annotators.\nSemi-supervised Dataset\n#\nKhatri et al. (2018)\nproposed a simple approach to bootstrap a large amount of semi-supervised dataset for learning toxic content classifiers. Their approach relies on a small annotated dataset and a large unlabelled dataset.\nFirst, they gather a blacklist of 800+ words covering topics of profanity, hate, sexual content and insults. A black list of profanities may have high precision and low recall, but it can provide weak supervised signals.\nSubreddits are sorted by the percentage of blacklisted words. Then sensitive examples are sampled from the top subreddits and non-sensitive ones from the bottom, respectively.\nTrain a weak binary classifier to further select more samples from the sorted subreddits,\nSensitive: contain blacklisted words or toxic classifier confidence > 0.8;\nNon-sensitive: not contain blacklisted words and toxic classifier confidence < 0.3\nGiven this large expanded dataset, train a new classifier named “Two-stage bootstrap” (\nTS bootstrap\n).\nTheir experiments showed that the TS bootstrap classifier achieved pretty good numbers on F1 score, accuracy and recall and it could also transfer to out-of-domain test data.\nThe two-stage bootstrap classifier is trained on a dataset bootstrapped by a weak toxic binary classifier on Reddit data. (Image source:\nKhatri et al. 2018\n)\nSOLID\n(Semi-Supervised Offensive Language Identification Dataset;\nRosenthal et al. 2020\n) contains 9+ M tweets annotated with the same taxonomy system as for\nOLID\n. SOLID treats OLID as a seed and extends it via a semi-supervised technique called\ndemocratic co-training\n. Democratic co-training (\nZhou & Goldman, 2004\n) creates a large dataset from noisy labels provided by a collection of diverse models trained on a small supervised dataset. SOLID is constructed by:\nFirst, train a diverse set of supervised models on the labeled dataset OLID. The paper experimented with PMI (n-gram-based similarity), FastText (shallow neural model similar to BoW model), LSTM and BERT.\nFor each sample in the unannotated dataset, each model predicts a confidence score for the target class. The scores are aggregated by taking\navg()\nor\nmin()\n. Samples with high confidence are added into the dataset.\nBERT model performance does not improve when the supervised dataset is large enough for a simple task, but can benefit from a big semi-supervised dataset if the original supervised dataset is too small for the task.\nToxicity Detection\n#\nGiven a supervised dataset, we can train a text classifier from scratch or fine-tune a pretrained language model to perform the classification task. But what if training samples are not good or sufficient enough? What if we don’t have access to such a supervised dataset?\nAdversarial Attacks\n#\nTo create a toxicity detection model that is robust to adversarial attacks,\nDinan et al. (2019)\nproposed an iterative “\nbuild it, break it, fix it\n” strategy to improve the dialogue system safety with humans in the loop.\nBuild it\n: A BERT model is trained to classify toxic comments on the\nJigsaw dataset\n.\nBreak it\n: Crowdsourced workers are asked to write toxic messages that are mistakenly labelled as “safe” by the model.\nFix it\n: The model is re-trained on the combination of the original dataset and newly collected adversarial samples.\nRepeat\n: Redeploy the robustified model and repeat a new round from step 1.\nThe illustration of iteratively improving a toxic content detection model via the \"build it, break it, fix it\" process. (Image source:\nDinan et al. 2019\n)\nOne baseline in their experiments is to replace the adversarial collection in the “break it” step with the standard collection where workers are asked to submit “offensive” messages directly . Compared to the standard collection, the adversarial collection has less explicit profanity and more negations to trick the model. The tasks become more challenging in the later rounds.\nAdversarial models are more robust against adversarial attacks than baseline models trained on the standard collection.  The third round adversarial model has worse performance on the standard task than the standard model, likely due to overfitting. I’m curious about how the model performance would be like if it is trained on both adversarial and standard collection, but I didn’t find it in the paper.\nThe comparison of performance on standard and adversarial tasks of models trained on standard ($S\\_i$) and adversarial data collection ($A\\_i$). The subscript $i$ indicates the number of training rounds. (Image source:\nDinan et al. 2019\n)\nAnother type of adversarial attack is to trick the detection model to mistakenly classify a toxic sentence as safe by replacing or scrambling a subset of characters.\nKurita et al. (2019)\ndeveloped a method of generating such model-agnostic adversarial attacks, incorporating several types of character-level perturbations:\nCharacter scrambling\n: randomly permute character positions.\nHomoglyph substitution\n: replace one or multiple letters with similar looking international letters.\nDictionary-based near-neighbor replacement\n: find closest but distinct token in terms of Levenshtein distance.\nDistractor injection\n: inject distractor tokens by repeating random selected sequences of non-toxic tokens.\nAdversarial noise combining token obfuscation and distractor tokens leads to substantial performance degradation of a toxic classifier. Character-level perturbation degrades performance more than distractors.\nThe paper proposed two ways to resolve adversarial attacks:\nAdversarial training\nrefers to training the model on a dataset with noise. However, you need to know the details of the incoming attacks in advance. And there is no guarantee that training samples with arbitrary noise would generalize to the test set.\nCDAE (contextual denoising autoencoder)\nuses character-level and contextual information to denoise obfuscated tokens. CDAE takes a noise sample to predict the denoised version. Still, you need to know what types of character-level perturbation can be applied to create noise samples. CDAE performs comparable to BERT, but not substantially better.\nPerspective API\n#\nperspective API\n(\nwww.perspectiveapi.com\n) is the most widely used commercial API for toxic content detection. Perspective trains machine learning models to provide scores for several different\nattributes\n: toxicity, severe toxicity, insult, profanity, identity attack, threat, and sexually explicit. Each score is a number between [0, 1], indicating how likely the message contains a given attribute (i.e. confidence of a binary classifier) and it does not signify the severity of the attribute.\nThe overview of Perspective API scores. (Image source:\nAbout Perspective API\n)\nGehman et al. (2020)\nmeasured the Perspective API toxicity scores of unprompted generations sampled from several pretrained language models. “Unprompted” means that the generation is only conditioned on the start-of-sentence tokens, without injecting any additional context. Noticeably, all the tested models get to the expected maximum toxicity > 0.5 after 100 generations. They also pointed out that training datasets for large LMs contain an non-negligible amount of toxic content.\nPerspective API toxicity scores of unprompted generations. Each model generates a pool of 10k samples and the expected maximum toxicity score is estimated via bootstrapping. (Image source:\nGehman et al. 2020\n)\nThey collected the\nRealToxicityPrompt\ndataset\nfor studying toxicity in conditional language model generation. It contains 100k naturally occurring prompts with associated toxicity scores from Perspective API. Some prompts that do not contain any toxic language still can trigger very offensive completion.\nDespite of its popularity, Perspective API contains known biases, as summarized by\nGehman et al. (2020)\n:\n… exhibit biases against minorities and suffer from low agreement in annotations, partially due to annotator identity influencing their perception of hate speech and differences in annotation task setup.\nNotably, recent work has found that systems are overestimating the prevalence of toxicity in text that contains a minority identity mention (e.g., “I’m a gay man”) or text by racial minorities (e.g., text in African American English). This is partially due to detectors’ over-reliance on lexical cues of toxicity (including swearwords, slurs, and other “bad” words).\nPrompt-based Detection\n#\nSelf-diagnosis\n(\nSchick, et al. 2021\n) is a process of exploiting the capacity of a pretrained language model to detect socially undesired attributes in its own outputs. The diagnosis depends on a predefined prompt template where the attributes are described in short text and measures the normalized probability of the model outputting “yes” versus “no”. Note that self-diagnosis does not need to access a labelled dataset for training.\n# Self-diagnosis template\n\"{sample text}\"\nQuestion: Does the above text contain {undesired attribute}?\nAnswer: {yes/no}\nThey use RealToxicityPrompts dataset and Perspective API for evaluation in the experiments. The self-diagnosis performance is positively correlated with the model size.\nSelf-diagnosis abilities for identifying undesired attributes. The ground truth is provided by Perspective API. (Image source:\nSchick, et al. 2021\n)\nDetoxification\n#\nBlacklisting\n#\nBad word filtering\nis a pretty intuitive and effective way to avoid explicit profane\nwords\nin the language model generation. At decoding time, we can manually reduce the probabilities of blocked words to avoid sampling them. However, it is not perfect, as it is still possible to have unsafe content composed of safe tokens.\nVocabulary shifting\n(\nGehman et al. 2020\n) learns a 2-dimensional representation of toxicity versus non-toxicity for every token in the vocabulary of the pretrained model. Then the representation that encodes the non-toxicity is used to boost the likelihood of non-toxic tokens at decoding time.\nPrompt-based Detox\n#\nSelf-debiasing\n(\nSchick et al. 2021\n) follows the similar idea as in\nself-diagnosis\n. It is a process for using the internal knowledge of a pretrained language model to reduce the probability of undesired attributes in the model generation.\n# Self-debiasing template, denoted as sdb(.)\nThe following text contains {undesired attribute s}:\n{sample text x}\nGiven an input prompt $\\mathbf{x}$, a textual description of undesired attributes $s$, and the language model $M$, self-debiasing computes the difference between the probability of next words without and with the self-debiasing template $\\text{sdb}(.)$:\n$$\n\\Delta(w, \\mathbf{x}, s) = p_M(w\\vert\\mathbf{x}) - p_M(w\\vert\\text{sdb}(\\mathbf{x}, s))\n$$\nBecause $\\text{sdb}(.)$ is expected to boost the probabilities of undesired words, $\\Delta(w, \\mathbf{x}, s)$ should be negative for undesirable words.\nIn self-diasing decoding, a scaling function of the probability difference $\\alpha(\\Delta(w, \\mathbf{x}, s)): \\mathbb{R}\\to[0,1]$ is used to alter the true sampling distribution,\n$$\n\\tilde{p}_M(w\\vert\\mathbf{x}) \\propto \\alpha(\\Delta(w, \\mathbf{x}, s)) p_M(w\\vert\\mathbf{x})\n$$\nIn the paper, they used a soft variant where the probabilities of the words with negative $\\Delta$ are reduced w.r.t. the magnitude of $\\Delta(w, \\mathbf{x}, s)$:\n$$\n\\alpha(x)=\\begin{cases} 1 & \\text{ if } x\\geq 0 \\\\ e^{\\lambda\\cdot x} & \\text{ otherwise} \\end{cases}\n$$\nSelf-diasing decoding can reduce the probabilities of undesirable attributes. The scores are provided by Perspective API. (Image source:\nSchick et al. 2021\n)\nThere are a couple of major limitations in self-debiasing detoxification:\nThe evaluation solely relies on Perspective API, so it cannot capture bias & toxicity attributes that are\nnot covered\nby Perspective API, such as gender biases. Using human evaluation is another alternative but the scale is limited.\nSelf-debiasing sometimes acts too aggressively and filters out harmless words and it does not maintain the same level of perplexity as the original model.\nThe approach is constrained by the internal capacity of the model. For example, if the model is not aware of certain biases, it would not be able to correct them.\nText Style Transfer\n#\nUnsupervised style transfer\ncan be used to translate offensive sentences into innocuous ones (\nSantos et al. 2018\n). The approach should work for non-parallel datasets, meaning that we only have access to two separate datasets of offensive and non-offensive samples, but not paired versions. To preserve the content when transferring the text into another style, a cycle consistency loss (\nZhu et al. 2017\n) is adopted.\nThe training process of a neural text style transfer algorithm using non-parallel data. (Image source:\nSantos et al. 2018\n)\nLet $s_i$ be the desired style ($i=0$ for offensive and $i=1$ for non-offensive), and $\\mathbf{x}^i_k$ be the $k$-th sample of style $s_i$, $k = 1, \\dots, n$. Both the encoder $E$ and decoder $G$ take a sample (or hidden state) along with a style label. The classifier $C$ predicts a probability distribution over the style labels given an input sample.\nFollowing the illustration in Fig. 9:\nThe top branch of forward transfer is auto encoder: ​$E(\\mathbf{x}^i_k, s_i) \\to H^i_k \\to G(H^i_k, s_i) \\to \\hat{\\mathbf{x}}^{i\\to i}_k$. Two losses are computed:\nReconstruction loss measures how well the decoder can reconstruct the sample back:\n$$\n    \\mathcal{L}_\\text{self} = \\mathbb{E}_{\\mathbf{x}^i_k \\sim \\mathcal{X}} [-\\log p_G(\\mathbf{x}_k^i \\mid E(\\mathbf{x}^i_k, s_i), s_i)]\n    $$\nThe bottom branch of forward transfer: $E(\\mathbf{x}^i_k, s_i) \\to H^i_k \\to G(H^i_k, s_j) \\to \\hat{\\mathbf{x}}^{i\\to j}_k$\nClassification loss measures the effectiveness of style transfer:\n$$\n    \\mathcal{L}_\\text{style_fwd} = \\mathbb{E}_{\\hat{\\mathbf{x}}^{i\\to j}_k \\sim \\hat{\\mathcal{X}}} [-\\log p_C(s_j \\mid \\hat{\\mathbf{x}}^{i\\to j}_k)]\n    $$\nThe back transfer uses cycle consistency loss: $E(\\hat{\\mathbf{x}}^{i\\to j}_k, s_j) \\to H^{i\\to j}_k \\to G(H^{i\\to j}_k, s_i) \\to \\hat{\\mathbf{x}}^{i\\to j \\to i}_k$\nThe cycle consistency loss controls how well the transferred sample can be converted back to the original form to encourage content preservation:\n$$\n    \\mathcal{L}_\\text{cycle} = \\mathbb{E}_{\\mathbf{x}^i_k \\sim \\mathcal{X}} [-\\log p_G(\\mathbf{x}_k^i \\mid E(\\hat{\\mathbf{x}}^{i \\to j}_k, s_j), s_i)]\n    $$\n- The classification loss ensures that the back-transferred sample has the correct label:\n$$\n    \\mathcal{L}_\\text{style_back} = \\mathbb{E}_{\\hat{\\mathbf{x}}^{i\\to j}_k \\sim \\hat{\\mathcal{X}}} [-\\log p_C(s_i \\mid G(E(\\hat{\\mathbf{x}}^{i\\to j}_k, s_j), s_i))]\n    $$\nThere is an additional supervised classification loss for training an accurate classifier:\n$$\n\\mathcal{L}_\\text{class} = \\mathbb{E}_{\\hat{\\mathbf{x}}^{i\\to j}_k \\sim \\hat{\\mathcal{X}}} [-\\log p_C(s_i \\mid \\hat{\\mathbf{x}}^i_k)]\n$$\nThe final training objective is as follows and the encoder, decoder and classifier are jointly trained:\n$$\n\\mathcal{L}(\\theta_E, \\theta_G, \\theta_C) = \\min_{E, G, C} \\mathcal{L}_\\text{self} + \\mathcal{L}_\\text{style_fwd} + \\mathcal{L}_\\text{cycle} + \\mathcal{L}_\\text{style_back}+ \\mathcal{L}_\\text{class}\n$$\nStyle Transformer\n(\nDai et al. 2019\n) also aims to learn unsupervised text style transfer. Different from the encoder-decoder model in\nSantos et al. 2018\n, it learns a Transformer-based style transfer function $f_\\theta(\\mathbf{x}, s)$ for a given input sample $\\mathbf{x}$ and a desired style control variable $s$.\nThe comparison of style transformer and previous models that depend on disentangled latent representation. (Image source:\nDai et al. 2019\n)\nWithout access to the parallel corpus, the style transformer adopts a discriminator to create supervision from non-parallel dataset.\nLet $s$ and $\\hat{s}$ be two mutually exclusive style variables and $\\mathbf{x}$ is a sample of style $s$, style transformer computes several losses:\nSelf reconstruction loss: $\\mathcal{L}_\\text{self} = - p_\\theta (\\mathbf{x} \\vert \\mathbf{x}, s)$\nCycle-consistency loss: $\\mathcal{L}_\\text{cycle} = - p_\\theta (\\mathbf{x} \\vert f_\\theta(\\mathbf{x}, \\hat{s}), s)$\nStyle controlling loss: This is necessary because otherwise the model would simply learn to copy the input over.\n$$\n\\mathcal{L}_\\text{style} = - p_\\phi(\\text{class} = 1 \\vert f_\\theta(\\mathbf{x}, \\hat{s}), \\hat{s})\n$$\n, where the discriminator is a simple binary classifier trained to optimize the negative log-likelihood of the correct style. The discriminator is trained by labelling\n$\\{(\\mathbf{x}, s), (f_\\theta(\\mathbf{x}, s), s), (f_\\theta(\\mathbf{x}, \\hat{s}), \\hat{s})\\}$ as positive class 1\n$\\{(\\mathbf{x}, \\hat{s}), (f_\\theta(\\mathbf{x}, s), \\hat{s}), (f_\\theta(\\mathbf{x}, \\hat{s}), s)\\}$ as negative class 0.\nThe training process of Style Transformer. (Image source:\nDai et al. 2019\n)\nDriven by the research question “Can we fine-tune a pre-trained language model to suggest civil rephrasings of rude comments using a dataset solely annotated in toxicity?”,\nLaugier et al. (2021)\nfine-tuned a pretrained text-to-text transformer with a denoising and cyclic auto-encoder loss.\nLet $s$ be the attribute of $\\mathbf{x}$ (e.g. “civil”) and $\\bar{s}$ be the other opposite attribute (e.g. “toxic”). These two attributes are mutually exclusive. The goal is to learn a mapping function $f_\\theta$ such that it translates $x$ to a new fluent sequence $y$ with target attribute $a$ while preserving $x$’s content.\nThe encoder-decoder model is trained with the loss:\n$$\n\\mathcal{L} = \\lambda_\\text{DAE} \\mathcal{L}_\\text{DAE} + \\lambda_\\text{cycle} \\mathcal{L}_\\text{cycle}\n$$\nThe denoising auto-encoder loss is the loss for denoising auto-encoders, where $\\eta$ is a\nmasking\nfunction same as in BERT training:\n$$\n\\mathcal{L}_\\text{DAE} = \\mathbb{E}_{\\mathbf{x} \\sim \\mathcal{X}} [−\\log p_\\theta(\\mathbf{x} \\mid \\eta(\\mathbf{x}), s)]\n$$\nThe cycle consistency loss (\nZhu et al. 2017\n) has $\\tilde{\\theta}$ to produce a non-differentiable pseudo-prediction $\\hat{\\mathbf{y}}$ and it does not take gradient backpropagation.\n$$\n\\mathcal{L}_\\text{cycle} = \\mathbb{E}_{\\mathbf{x} \\sim \\mathcal{X}} [−\\log p_\\theta(\\mathbf{x} \\mid f_{\\tilde{\\theta}}(\\mathbf{x}, \\bar{s}), s)]\n$$\nThey used the above loss to fine-tune a T5 model, resulting in a model named\nCAE-T5\n. The conditioning is implemented like CTRL via control code (“civil” or “toxic”) prepended to the start of a sequence.\nAutomatic evaluation of the text style transferred results relies on three metrics:\nAccuracy\n: Classification accuracy measures how successful the style transfer is.\nFluency\n: Fluency is commonly measured by perplexity by another separately trained LM on non-toxic samples.\nContent preservation\n: It is the content similarity between transferred and original sentences, measured by BLEU or embedding based content similarity.\nHuman evaluation is also necessary but more costly.\nCompared to the baseline (\nShen et al. 2017\n), the style transfer method by\nSantos et al. 2018\nachieves better classification accuracy, better content preservation, but worse perplexity. CAE-T5 has worse classification accuracy, competitive content preservation, and better perplexity compared to a set of baselines including Style Transformer.\nControllable Generation\n#\nWe can try to avoid toxic outputs via\ncontrollable text generation\n. There are several popular approaches for steering a pretrained language model toward desired styles, topics or safety criteria:\nApply guided decoding strategies and select desired outputs at test time.\nOptimize for the most desired outcomes via good prompt design.\nFine-tune the base model or steerable layers to do conditioned content generation.\nRead more in my\nlast post\non controllable neural text generation, introducing methods like\nAutoPrompt\n,\nCTRL\n,\nPPLM\n,\nGeDi\nand many more.\nGehman et al. (2020)\nexperimented with both data-based (supervised fine-tuning, CTRL training) and decoding-based (vocabulary shifting, blocked word filtering, PPLM) methods for language model detoxification. They found that toxicity control tokens (CTRL) and swear word filters are\nless successful\nthan more computationally or data-intensive methods like fine-tuning on non-toxic corpora and PPLM.\nTable list expected maximum toxicity score over 25 generations (left) and the empirical probability of generating toxic text over 25 generations (right) for several detoxification methods. Scores are provided by Perspective API. (Image source:\nGehman et al., 2020\n)\nSystem-level Safety Solution\n#\nXu et al. (2020)\npresented a thorough system-level design for building safe chatbots.\nIllustration of a safe chat bot system. (Image source:\nXu et al. 2020\n)\nThey consider four general strategies in the recipes for making the bot safer:\nDetect unsafe content\n: Adopt a classifier for detecting unsafe language on both the input and output side, as an extra safety layer on top of the language model.\nThe classifier is trained on an enhanced version of the\nJigsaw toxic\ncomment dataset (safe vs unsafe binary labels), extended with\nadversarial human attacks\n(\nDinan et al. 2019\n) and\nsemi-supervision\n(\nKhatri et al. 2018\n).\nThe safety classifier can be used on both the user input and the model output. If it detects unsafe content, the system is configured to return a canned, predefined response (e.g “I’m sorry I’m not sure what to say.”), or decide to change topics. It is worthy noting that this approach relies on a high-quality classifier. The conversation experience would be drastically disrupted with too many false positives.\nBot adversarial dialogue (BAD) safety: The idea is to collect data on humans adversarially probing the system to make mistakes and then use the data for further training. During annotation, human labellers can tag the bot’s response with an unsafe-safe rating based on the percentage of population who may consider it as unsafe. This probing data collection is used to train a multi-turn safety classifier, predicting whether a response is offensive given the dialogue context.\nSafe generation\n: Train a model that is less likely to output unsafe responses.\nA predefined list of unsafe words/n-grams can be\nblocked\nat decoding time.\nThe pretraining data is filtered by the above safety classifier, or filtered based on known authors.\nThe problem with pre-training only with safe datasets is that if the model has never seen toxic language during training, it would not know how to respond at test time (OOD; e.g. may just copy the offensive content). They instead prepare a collection of training samples where the last utterance is labelled as “unsafe” and then attach a safe response following that unsafe attack. Then the model is fine-tuned on the “baked-in” safety data.\nDo\nCTRL\nstyle training by assigning “safe” vs “unsafe” label using the safety classifier.\nAvoid sensitive topics\n:\nIn order to avoid sensitive topics (politics, religion, drug use, medical advice, and NSFW and relationships/dating), they trained a multi-class classifier to detect those topics using crowdsourced lists of subreddits. The classifier can be periodically re-trained to capture the changes within topics over time.\nA small validation set is collected by recruiting crowdsourced workers to discuss one of the target topics.\nGender bias mitigation\n:\nThey used\nCTRL\nstyle training to mitigate gender biases.\nPrecisely, given a gendered word list, tag the training samples with $F^0 M^0$, $F^0 M^+$, $F^+ M^+$, and $F^+ M^0$ labels, indicating whether the response contains female / male words ($+$ contains, $-$ does not contain). At test time, the system runs with a control label $F^0 M^0$ to avoid outputting gender specific words.\nAppendix: Datasets\n#\n(*Only datasets in English are listed here.)\nHate Speech and Offensive Language\nDataset (2017): contains about 25k tweets, each labelled manually as one of three categories: hate speech, offensive but not hate speech, or neither offensive nor hate speech. [\nDownload\n]\nJigsaw Toxic\nComments Classification Dataset (2018): contains about 160k examples extracted from Wikipedia discussion pages, each annotated for 7 classes: toxic, severe toxic, obscene, threat, insult, identity hate and non-toxic. The labelling process involved 5000 crowdsourced annotators. [\nDownload\n]\nJigsaw Unintended Bias in Toxicity\nClassification Dataset (2019): contains about 2 Millions comments from the Civil Comments platform, which shut down in 2017. This data is annotated for toxicity, toxicity sub-types, and mentions of identities, which enables evaluation of unintended bias with respect to identity mentions. [\nDownload\n]\nOLID\n(Offensive Language Identification Dataset; 2019): contains 14,100 English tweets, annotated according to the three-level taxonomy as described\nhere\n. [\nDownload\n]\nSOLID\n(Semi-Supervised Offensive Language Identification Dataset; 2020): contains 9+ Millions tweets annotated following OLID’s three level taxonomy. [\nDownload\n]\nRealToxicityPrompts\ndataset (2020): contains 100k sentence snippets from the web with Perspective API toxicity scores for studying the risk of neural toxic degeneration in language models. [\nDownload\n]\nCitation\n#\nCited as:\nWeng, Lilian. (Mar 2021). Reducing toxicity in language models. Lil’Log. https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/.\nOr\n@article{weng2021toxic,\n  title   = \"Reducing Toxicity in Language Models.\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2021\",\n  month   = \"Mar\",\n  url     = \"https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/\"\n}\nReferences\n#\n[1] Vidgen, et al.\n“Challenges and frontiers in abusive content detection.”\nWorkshop on Abusive Language Online 2019.\n[2] Zampieri et al.\n“Predicting the type and target of offensive posts in social media.”\nNAACL 2019.\n[3] Vidgen & Deczynski.\n“Directions in abusive language training data, a systematic review: Garbage in, garbage out.”\nPLoS ONE 15(12): e0243300 (2020).\n[4] Davidson et al.\n“Automated hate speech detection and the problem of offensive language.”\nICWSM 2017.\n[5] Khatri et al.\n“Detecting offensive content in open-domain conversations using two stage semi-supervision.”\nNeuriIPS CONVAI Workshop 2018.\n[6] Rosenthal et al.\n“A Large-Scale Semi-Supervised Dataset for Offensive Language Identification”\narXiv:2004.14454 (2020).\n[7] Pavlopoulos et al.\n“Toxicity Detection: Does Context Really Matter?”\narXiv:2006.00998 (2020).\n[8] Dinan et al.\n“Build it, break it, fix it for dialogue safety: Robustness from adversarial human attack.”\narXiv:1908.06083 (2019).\n[9] Kurita et al.\n“Towards Robust Toxic Content Classification”\narXiv:1912.06872 (2019)\n[10] Santos et al.\n“Fighting offensive language on social media with unsupervised text style transfer.”\narXiv:1805.07685 (2018)\n[11] Dai et al.\n“Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation”\nACL 2019.\n[12] Laugier et al.\n“Civil Rephrases Of Toxic Texts With Self-Supervised Transformers”\narXiv:2102.05456 (2021).\ncode\n[13] Schick et al.\n“Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP”\narXiv:2103.00453 (2021).\n[14] Gehman et al.\n“RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models”\nEMNLP 2020.\n[15] Xu et al.\n“Recipes for Safety in Open-domain Chatbots”\narXiv:2010.07079 (2020).",
  "metadata": {
    "title": "Reducing Toxicity in Language Models | Lil'Log",
    "description": "Large pretrained language models are trained over a sizable collection of online data. They unavoidably acquire certain toxic behavior and biases from the Internet. Pretrained language models are very powerful and have shown great success in many NLP tasks. However, to safely deploy them for practical real-world applications demands a strong safety control over the model generation process.",
    "keywords": "nlp, language-model, alignment, steerability, safety",
    "author": "Lilian Weng",
    "og:title": "Reducing Toxicity in Language Models",
    "og:description": "Large pretrained language models are trained over a sizable collection of online data. They unavoidably acquire certain toxic behavior and biases from the Internet. Pretrained language models are very powerful and have shown great success in many NLP tasks. However, to safely deploy them for practical real-world applications demands a strong safety control over the model generation process.",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/",
    "twitter:card": "summary",
    "twitter:title": "Reducing Toxicity in Language Models",
    "twitter:description": "Large pretrained language models are trained over a sizable collection of online data. They unavoidably acquire certain toxic behavior and biases from the Internet. Pretrained language models are very powerful and have shown great success in many NLP tasks. However, to safely deploy them for practical real-world applications demands a strong safety control over the model generation process.",
    "article:section": "posts",
    "article:published_time": "2021-03-21T00:00:00+00:00",
    "article:modified_time": "2021-03-21T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/5bdaf053b66c/code/body-000.txt",
      "chars": 120,
      "language": "text"
    },
    {
      "file": "attachments/5bdaf053b66c/code/body-001.txt",
      "chars": 114,
      "language": "text"
    },
    {
      "file": "attachments/5bdaf053b66c/code/body-002.txt",
      "chars": 90,
      "language": "text"
    },
    {
      "file": "attachments/5bdaf053b66c/code/body-003.txt",
      "chars": 255,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/offensive-taxonomy.png",
      "bytes": 261312,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-7912a5dbfe2f7af86328.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/offensive-taxonomy.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/TS-bootstrap.png",
      "bytes": 240131,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-911cb6f1748d054d5fb2.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/TS-bootstrap.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/build-break-fix.png",
      "bytes": 363472,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-0b1801633c8070fdb0bd.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/build-break-fix.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/build-break-fix-it-results.png",
      "bytes": 115281,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-336d687a931cffb58c4a.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/build-break-fix-it-results.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/about-perspective-api.png",
      "bytes": 115123,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-890d5bc60c0026eacf48.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/about-perspective-api.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/unprompted-toxicity.png",
      "bytes": 716772,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-8a35c9808d73cd898bb2.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/unprompted-toxicity.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/self-diagnosis-toxicity-score.png",
      "bytes": 384801,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-80aad0215234f5577ca9.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/self-diagnosis-toxicity-score.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/self-debiasing-decoding.png",
      "bytes": 133159,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-1b1f9b33fdf91ce3e874.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/self-debiasing-decoding.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/offensive-text-style-transfer.png",
      "bytes": 297398,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-e8d007598b31fe345db4.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/offensive-text-style-transfer.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/style-transformer.png",
      "bytes": 223977,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-560bcbb86d5a49d50843.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/style-transformer.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/style-transformer-training.png",
      "bytes": 228005,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-43195c87d2fdc1542dbb.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/style-transformer-training.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/RealToxicityPrompts-experiments.png",
      "bytes": 197132,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-b6bf808e00719a46b6e3.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/RealToxicityPrompts-experiments.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/safe-chatbot-system.png",
      "bytes": 247202,
      "reused": false,
      "local_path": "attachments/5bdaf053b66c/images/body-4254e21dbc5919fae389.png",
      "src": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/safe-chatbot-system.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://lilianweng.github.io/posts/2019-01-31-lm/",
      "label": "language models",
      "type": "a"
    },
    {
      "href": "https://support.perspectiveapi.com/s/about-the-api-attributes-and-languages",
      "label": "Perspective API",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1912.06872",
      "label": "Kurita et al. 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.00998",
      "label": "Pavlopoulos et al. 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1902.09666",
      "label": "Zampieri et al. (2019)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2004.01670",
      "label": "Vidgen & Derczynski (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1703.04009",
      "label": "Davidson et al. 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1811.12900",
      "label": "Khatri et al. (2018)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2004.14454",
      "label": "Rosenthal et al. 2020",
      "type": "a"
    },
    {
      "href": "https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.76.3152&rep=rep1&type=pdf",
      "label": "Zhou & Goldman, 2004",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1908.06083",
      "label": "Dinan et al. (2019)",
      "type": "a"
    },
    {
      "href": "https://www.perspectiveapi.com/",
      "label": "www.perspectiveapi.com",
      "type": "a"
    },
    {
      "href": "https://support.perspectiveapi.com/s/about-the-api",
      "label": "About Perspective API",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.11462",
      "label": "Gehman et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2103.00453",
      "label": "Schick, et al. 2021",
      "type": "a"
    },
    {
      "href": "https://github.com/%20LDNOOBW/List-of-Dirty-Naughty-Obscene-%20and-Otherwise-Bad-Words",
      "label": "words",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1805.07685",
      "label": "Santos et al. 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1703.10593",
      "label": "Zhu et al. 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.05621",
      "label": "Dai et al. 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2102.05456",
      "label": "Laugier et al. (2021)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1705.09655",
      "label": "Shen et al. 2017",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/",
      "label": "last post",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.15980",
      "label": "AutoPrompt",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1909.05858",
      "label": "CTRL",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1912.02164",
      "label": "PPLM",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.06367",
      "label": "GeDi",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.07079",
      "label": "Xu et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://github.com/t-davidson/hate-speech-and-offensive-language/blob/master/data/readme.md",
      "label": "Download",
      "type": "a"
    },
    {
      "href": "https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge",
      "label": "Download",
      "type": "a"
    },
    {
      "href": "https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification",
      "label": "Download",
      "type": "a"
    },
    {
      "href": "https://sites.google.com/site/offensevalsharedtask/olid",
      "label": "Download",
      "type": "a"
    },
    {
      "href": "https://sites.google.com/site/offensevalsharedtask/solid",
      "label": "Download",
      "type": "a"
    },
    {
      "href": "https://allenai.org/data/real-toxicity-prompts",
      "label": "Download",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/W19-3509/",
      "label": "“Challenges and frontiers in abusive content detection.”",
      "type": "a"
    },
    {
      "href": "https://github.com/LeoLaugier/conditional-auto-encoder-text-to-text-transfer-transformer",
      "label": "code",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:26Z",
  "source_url": "https://lilianweng.github.io/posts/2021-03-21-lm-toxicity/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1912.06872",
      "bytes": 1190588,
      "reused": false,
      "local_path": "papers/repair/1933d921b63dacf16849.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.00998",
      "bytes": 375324,
      "reused": false,
      "local_path": "papers/repair/1f200ab2f468e2413465.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1902.09666",
      "bytes": 101387,
      "reused": false,
      "local_path": "papers/repair/8f51d5af804a0ad30b34.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2004.01670",
      "bytes": 623777,
      "reused": false,
      "local_path": "papers/repair/0d11d9f063fa43d2600b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.04009",
      "bytes": 182413,
      "reused": false,
      "local_path": "papers/repair/1c692e7e385ba4cfdae1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1811.12900",
      "bytes": 390417,
      "reused": false,
      "local_path": "papers/repair/7336a57c0b8ecd68f306.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2004.14454",
      "bytes": 343418,
      "reused": false,
      "local_path": "papers/repair/f69b585200c8e8a5611f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1908.06083",
      "bytes": 461117,
      "reused": false,
      "local_path": "papers/repair/558e5046d938c9ce186d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.11462",
      "bytes": 1046316,
      "reused": false,
      "local_path": "papers/repair/cfec2fa4accdf46e1292.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2103.00453",
      "bytes": 458920,
      "reused": false,
      "local_path": "papers/repair/6367e717da882e46264d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1805.07685",
      "bytes": 402611,
      "reused": false,
      "local_path": "papers/repair/fac7a71dbb401f70b2f2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1703.10593",
      "bytes": 37551521,
      "reused": false,
      "local_path": "papers/repair/734bdd7e1bbcb1441b48.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.05621",
      "bytes": 351366,
      "reused": false,
      "local_path": "papers/repair/cd633ef0e7ee73567e35.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2102.05456",
      "bytes": 876247,
      "reused": false,
      "local_path": "papers/repair/f85ead647e9ae8ed33a0.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1705.09655",
      "bytes": 375296,
      "reused": false,
      "local_path": "papers/repair/105e83ab72cdf291ee1c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.15980",
      "bytes": 663180,
      "reused": true,
      "local_path": "papers/repair/5fc241cafafb7eeca9db.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1909.05858",
      "bytes": 397593,
      "reused": true,
      "local_path": "papers/repair/88b09dc46a4672339e3b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1912.02164",
      "bytes": 589474,
      "reused": false,
      "local_path": "papers/repair/90a96ef437071e1f745a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.06367",
      "bytes": 1361683,
      "reused": false,
      "local_path": "papers/repair/5368b60783b2cc11e761.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.07079",
      "bytes": 1675082,
      "reused": true,
      "local_path": "papers/repair/0055b90c4e70b6f3c415.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 32730,
  "expected_body_images": 13,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/5bdaf053b66c/pages/source.html",
  "local_html": "attachments/5bdaf053b66c/pages/article.html",
  "body_base_path": "attachments/5bdaf053b66c/pages/",
  "body_markdown_path": "attachments/5bdaf053b66c/pages/article.md"
}