{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "Controllable Neural Text Generation",
  "keywords": "可控文本生成,CTRL,解码",
  "domain": "文本处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/",
  "publish_date": "2021-01",
  "corpus_stage": "预训练/SFT",
  "notes": "可控文本生成综述",
  "id": "91c8f7fce27f",
  "collected_at": "2026-10-02T14:13:08",
  "title": "Controllable Neural Text Generation | Lil'Log",
  "body_markdown": "[Updated on 2021-02-01: Updated to version 2.0 with several work added and many typos fixed.]\n[Updated on 2021-05-26: Add P-tuning and Prompt Tuning in the\n“prompt design”\nsection.]\n[Updated on 2021-09-19: Add\n“unlikelihood training”\n.]\nThere is a gigantic amount of free text on the Web, several magnitude more than labelled benchmark datasets. The state-of-the-art language models (LM) are trained with unsupervised Web data in large scale. When generating samples from LM by iteratively sampling the next token, we do not have much control over attributes of the output text, such as the topic, the style, the sentiment, etc. Many applications would demand a good control over the model output. For example, if we plan to use LM to generate reading materials for kids, we would like to guide the output stories to be safe, educational and easily understood by children.\nHow to steer a powerful unconditioned language model? In this post, we will delve into several approaches for controlled content generation with an unconditioned langage model.\nNote that model steerability is still an open research question. Each introduced method has certain pros & cons.\nApply guided decoding strategies and select desired outputs at test time.\nOptimize for the most desired outcomes via good prompt design.\nFine-tune the base model or steerable layers to do conditioned content generation.\nIn the following discussion, we assume we have access to a pretrained generative language model $p_\\theta$. The model has learned the distribution over token sequences by optimizing for the next token prediction: $ \\mathcal{L}_\\text{ML} = - \\sum_t \\log p_\\theta(x_t \\vert x_{<t}) $.\nDecoding Strategies\n#\nBy adopting different decoding methods, we can place restrictions or preferences on the sampling process to alter the generated samples without modifying any model weights. Even though decoding strategies do not change the values of any trainable parameter, it is a quite important component.\nCommon Decoding Methods\n#\nSince the final layer of the model predicts logits $o$ over the vocabulary space, the next token can be sampled by applying softmax with temperature $T$. The probability of sampling the $i$-th token is\n$$\np_i \\propto \\frac{\\exp(o_i / T)}{\\sum_j \\exp(o_j/T)}\n$$\nA low temperature would make the distribution sharper and a high value makes it softer.\nGreedy search\n: Always pick the next token with the\nhighest\nprobability, equivalent to setting temperature $T=0$. However, it tends to create repetitions of phrases, even for well-trained models.\nBeam search\n: It essentially does breadth-first search, one token per tree level, but with a limited bandwidth. At each level of the search tree, beam search keeps track of $n$ (named “beam width”) best candidates and expands all the successors of these candidates in the next level. Beam search could stop expanding a node if it hits the EOS (end-of-sentence) token.\nHowever, maximization-based decoding does not guarantee high-quality generation.\nThe probability assigned to the next token by beam search versus by humans. The human selected tokens have much higher variance in predicted probability and thus more surprising. (Image source:\nHoltzman et al. 2019\n)\nTop-k sampling\n(\nFan et al., 2018\n): At each sampling step, only the top $k$ most likely tokens are selected and the probability mass is redistributed among them. In\nFan et al., 2018\n, the authors proposed to use\ntop-k random sampling\nwhere the next token is randomly selected among the top $k$ most likely candidates and they argued that this approach can generate more novel and less repetitive content than beam search.\nNucleus sampling\n(\nHoltzman et al. 2019\n): Also known as “Top-p sampling”. One drawback of top-k sampling is that the predefined number $k$ does not take into consideration how\nskewed\nthe probability distribution might be. The nucleus sampling selects the smallest set of top candidates with the cumulative probability exceeding a threshold (e.g. 0.95) and then the distribution is rescaled among selected candidates.\nBoth top-k and nucleus sampling have less repetitions with a proper set of hyperparameters.\nPenalized sampling\n(\nKeskar et al. 2019\n): To avoid the common failure case of generating duplicate substrings, the\nCTRL\npaper proposed a new sampling method to penalize repetitions by discounting the scores of previously generated tokens. The probability distribution for the next token with repetition penalty is defined as:\n$$\np_i = \\frac{\\exp(o_i / (T \\cdot \\mathbb{1}(i \\in g)))}{\\sum_j \\exp(o_j / (T \\cdot \\mathbb{1}(j \\in g)))} \\quad\n\\mathbb{1}(c) = \\theta \\text{ if the condition }c\\text{ is True else }1\n$$\nwhere $g$ contains a set of previously generated tokens, $\\mathbb{1}(.)$ is an identity function. $\\theta=1.2$ is found to yield a good balance between less repetition and truthful generation.\nGuided Decoding\n#\nAll the above standard decoding strategies sample tokens according to the predicted probability, with no additional information. Our preferences on topic or sentiment can be baked into the candidate ranking function to guide the sample generation by altering the candidate ranking score. The ranking score for token selection at each decoding step can be set as a combination of LM log-likelihood and a set of desired feature discriminators. The features are designed to quantify human preferences by heuristics (\nGhazvininejad et al., 2017\n), supervised learning (\nHoltzman et al., 2018\n) or RL (\nLi et al., 2017\n).\nGhazvininejad et al. (2017)\nbuilt a system called “Hafez” for generating poetry in desired style by adjusting sampling weights in beam search at decoding steps. The likelihood of sampling for the next token $x_{t+1}$ at step $t$ is augmented by a scoring function:\n$$\n\\text{score}(x_{t+1}, b_t) = \\text{score}(b_t) + \\log p(x_{t+1}) + \\color{green}{\\sum_i \\alpha_i f_i(x_{t+1})}\n$$\nwhere $\\log p(x_{t+1})$ is the log-likelihood predicted by LM. $\\text{score}(b_t)$ is the accumulated score of the already-generated words in the current beam state $b_t$. The green part can incorporate many different features for steering the style of the output. A set of feature functions $f_i(.)$ define the preferences and the associated weights $alpha_i$ work like “control knobs” that can be easily customized at decoding time. Features can measure a variety of attributes and can be easily combined; for example,\nwhether $x_{t+1}$ exists in a bag of desired or banned topical words.\nwhether $x_{t+1}$ indicates certain sentiments.\nwhether $x_{t+1}$ is a repeated token (and thus $f_i$ needs to take the history as input too).\nthe length of $x_{t+1}$ if longer or shorter words are in particular preferred.\nSimilar to Hafez,\nBaheti et al. (2018)\nmanually designed features for ranking and altered the sampling distribution by appending similarity scores between topic distribution or embeddings of the context and the completion.\nHoltzman et al. (2018)\nadopted a set of learned discriminators, each specializing in a different principle of communication guided by\nGrice’s maxims\n: quality, quantity, relation and manner. The discriminators learn to encode these desired principles by measuring repetition, entailment, relevance, and lexical diversity, respectively. Given some ground truth completion, all the discriminator models are trained to minimize the ranking log-likelihood, $\\log\\sigma(f_i(y_g) - f_i(y))$, because the gold continuation $y_g$ is expected to obtain a higher score than the generated one $y$. Here the weight coefficients $\\alpha_i$ are also learned to minimize the score difference between the golden standard and the generated completion.  Discriminative Adversarial Search (DAS;\nScialom et al., 2020\n) is inspired by GAN and trains the discriminator to tell apart human created text from machine generated text. The discriminator predicts a label for each token instead of for the entire sequence. The discriminator logprob is added to the score to guide sampling towards the human-written style.\nMeister et al. (2020)\nstudied beam search in a regularized decoding framework:\n$$\n\\mathbf{y}^* = \\arg\\max_{\\mathbf{y}\\in\\mathcal{Y}} \\big( \\underbrace{\\log p_\\theta(\\mathbf{y}\\vert\\mathbf{x})}_\\text{MAP} - \\underbrace{\\lambda\\mathcal{R}(\\mathbf{y})}_\\text{regularizer} \\big)\n$$\nSince we expect maximum probability to have minimum surprise, the surprisal of a LM at time step $t$ can be defined as follows:\n$$\n\\begin{aligned}\nu_0(\\texttt{BOS}) &= 0 \\text{  ; BOS is a placeholder token for the beginning of a sentence.}\\\\\nu_t(y) &= -\\log P_\\theta(y \\vert \\mathbf{x}, \\mathbf{y}_{<{t}}) \\text{ for }t \\geq 1\n\\end{aligned}\n$$\nThe MAP (maximum a posteriori) part demands for sequences with maximum probability given context, while the regularizer introduces other constraints. It is possible a global optimal strategy may need to have a high-surprisal step occasionally so that it can shorten the output length or produce more low-surprisal steps afterwards.\nBeam search has gone through the test of time in the field of NLP. The question is:\nIf we want to model beam search as exact search in a regularized decoding framework, how should $\\mathcal{R}(\\mathbf{y})$ be modeled?\nThe paper proposed a connection between beam search and the\nuniform information density\n(UID) hypothesis.\n“The uniform information density hypothesis (UID; Levy and Jaeger, 2007)  states that—subject to the constraints of the grammar—humans prefer sentences that distribute information (in the sense of information theory) equally across the linguistic signal, e.g., a sentence.”\nIn other words, it hypothesizes that humans prefer text with evenly distributed surprisal. Popular decoding methods like top-k sampling or nuclear sampling actually filter out high-surprisal options, thus implicitly encouraging the UID property in output sequences.\nThe paper experimented with several forms of regularizers:\nGreedy\n: $\\mathcal{R}_\\text{greedy}(\\mathbf{y}) = \\sum_{t=1}^{\\vert\\mathbf{y}\\vert} \\big(u_t(y_t) - \\min_{y’ \\in \\mathcal{V}} u_t(y’) \\big)^2$; if set $\\lambda \\to \\infty$, we have greedy search. Note that being greedy at each individual step does not guarantee global optimality.\nVariance regularizer\n: $\\mathcal{R}_\\text{var}(\\mathbf{y}) = \\frac{1}{\\vert\\mathbf{y}\\vert}\\sum_{t=1}^{\\vert\\mathbf{y}\\vert} \\big(u_t(y_t) - \\bar{u} \\big)^2$ , where $\\bar{u}$ is the average surprisal over all timesteps. It directly encodes the UID hypothesis.\nLocal consistency\n: $\\mathcal{R}_\\text{local}(\\mathbf{y}) = \\frac{1}{\\vert\\mathbf{y}\\vert}\\sum_{t=1}^{\\vert\\mathbf{y}\\vert} \\big(u_t(y_t) - u_{t-1}(y_{t-1}) \\big)^2$; this decoding regularizer encourages adjacent tokens to have similar surprisal.\nMax regularizer\n: $\\mathcal{R}_\\text{max}(\\mathbf{y}) = \\max_t u_t(y_t)$ penalizes the maximum compensation of surprisal.\nSquared regularizer\n: $\\mathcal{R}_\\text{square}(\\mathbf{y}) = \\sum_{t=1}^{\\vert\\mathbf{y}\\vert} u_t(y_t)^2$ encourages all the tokens to have surprisal close to 0.\nAn experiment with greedy regularizers showed that larger $\\lambda$ results in better performance (e.g. measured by BLEU for NMT task) and lower std dev of surprisal.\nThe plot of BLEU and std. dev of surprisals as functions of the strength of the regularizer $\\lambda$. The subgraph in grey shows the relationship between BLEU and surprisal std. dev. (Image source:\nMeister et al. 2020\n)\nA default beam search would have text generation of decreased quality when beam size increases. Regularized beam search greatly helps alleviate this issue. A combined regularizer further improves the performance. In their experiments for NMT, they found $\\lambda=5$ for greedy and $\\lambda=2$ for squared work out as the optimal combined regularizer.\nThe plot of BLEU of a function of beam size (left) and BLEU scores for translations created by different regularized decoding strategies. (Image source:\nMeister et al. 2020\n)\nGuided decoding essentially runs a more expensive beam search where the sampling probability distribution is altered by side information about human preferences.\nTrainable Decoding\n#\nGiven a trained language model,\nGu et al (2017)\nproposed a\ntrainable greedy decoding\nalgorithm to maximize an arbitrary objective for sampling sequences. The idea is based on the\nnoisy, parallel approximate decoding\n(\nNPAD\n). NPAD injects unstructured noise into the model hidden states and runs noisy decoding multiple times in parallel to avoid potential degradation. To take a step further, trainable greedy decoding replaces the unstructured noise with a learnable random variable, predicted by a RL agent that takes the previous hidden state, the previous decoded token and the context as input. In other words, the decoding algorithm learns a RL actor to manipulate the model hidden states for better outcomes.\nGrover et al. (2019)\ntrained a binary classifier to distinguish samples from data distribution and samples from the generative model. This classifier is used to estimate\nimportance weights\nfor constructing a new unnormalized distribution. The proposed strategy is called\nlikelihood-free importance weighting (LFIW)\n.\nLet $p$ be the real data distribution and $p_\\theta$ be a learned generative model. A classical approach for evaluating the expectation of a given function $f$ under $p$ using samples from $p_\\theta$ is to use importance sampling.\n$$\n\\mathbb{E}_{\\mathbf{x}\\sim p} [f(\\mathbf{x})] \n= \\mathbb{E}_{\\mathbf{x}\\sim p_\\theta} \\Big[\\frac{p(\\mathbf{x})}{p_\\theta(\\mathbf{x})} f(\\mathbf{x})\\Big]\n\\approx \\frac{1}{N} \\sum_{i=1}^N w(\\mathbf{x}_i)f(\\mathbf{x}_i)\n$$\nHowever, $p(\\mathbf{x})$ can only be estimated via finite datasets. Let $c_\\phi: \\mathcal{X} \\to [0,1]$ be a probabilistic binary classifier for predicting whether a sample $\\mathbf{x}$ is from the true data distribution ($y=1$). The joint distribution over $\\mathcal{X}\\times\\mathcal{Y}$ is denoted as $q(\\mathbf{x}, y)$.\n$$\nq(\\mathbf{x}\\vert y) = \\begin{cases}\np_\\theta(\\mathbf{x}) & \\text{ if }y=0\\text{; predicted to be generated data} \\\\\np(\\mathbf{x}) & \\text{ otherwise; from the true data distribution}\n\\end{cases}\n$$\nThen if $c_\\phi$ is\nBayes optimal\n, the importance weight can be estimated by:\n$$\nw_\\phi(\\mathbf{x}) \n= \\frac{p(\\mathbf{x})}{p_\\theta(\\mathbf{x})}\n= \\frac{q(\\mathbf{x} \\vert y=1)}{q(\\mathbf{x} \\vert y=0)}\n= \\frac{q(y=0)}{q(y=1)} \\frac{q(y=1 \\vert \\mathbf{x})}{q(y=0 \\vert \\mathbf{x})}\n= \\gamma \\frac{c_\\phi(\\mathbf{x})}{1 - c_\\phi(\\mathbf{x})}\n$$\nwhere $\\gamma = \\frac{q(y=0)}{q(y=1)} > 0$ is a fixed odd ratio.\nSince we cannot learn a perfect optimal classifier, the importance weight would be an estimation $\\hat{w}_\\phi$. A couple of practical tricks can be applied to offset cases when the classifier exploits artifacts in the generated samples to make very confident predictions (i.e. very small importance weights):\nSelf-normalization: normalize the weight by the sum $\\hat{w}_\\phi(\\mathbf{x}_i) / \\sum_{j=1}^N \\hat{w}_\\phi(\\mathbf{x}_j)$.\nFlattening: add a power scaling parameter $\\alpha > 0$, $\\hat{w}_\\phi(\\mathbf{x}_i)^\\alpha$.\nClipping: specify a lower bound $\\max(\\hat{w}_\\phi(\\mathbf{x}_i), \\beta)$.\nTo sample from an importance resampled generative model, $\\mathbf{x}\\sim p_{\\theta, \\phi}(\\mathbf{x}) \\propto p_\\theta(\\mathbf{x})\\hat{w}_\\phi(\\mathbf{x})$, they adopt SIR (Sampling-Importance-Resampling),\nThe algorithm for sampling from a generative model according to importance weights $\\hat{w}(\\mathbf{x}\\_i)$ using SIR. (Image source:\nGrover et al., 2019)\n)\nDeng et al., 2020\nproposed to learn a EBM to steer a LM in the\nresidual space\n, $P_\\theta(x) \\propto P_\\text{LM}(x)\\exp(-E_\\theta(x))$, where $P_\\theta$ is the joint model; $E_\\theta$ is the residual energy function to be learned. If we know the partition function $Z$, we can model the generative model for generative a sequence $x_{p+1}, \\dots, x_T$ as:\n$$\nP_\\theta(x_{p+1:T}\\vert x_{1:p}) = \\frac{P_\\text{LM}(x_{p+1:T}\\vert x_{1:p}) \\exp(-E_\\theta(x_{1:T}))}{Z_\\theta(x_{1:p})}\n$$\nThe goal is to learn the parameters of the energy function $E_\\theta$ such that the joint model $P_\\theta$ gets closer to the desired data distribution. The residual energy function is trained by noise contrastive estimation (\nNCE\n), considering $P_\\theta$ as the model distribution and $P_\\text{LM}$ as the noise distribution:\n$$\n\\theta = \\arg\\max_{\\theta} \\mathbb{E}_{x^+ \\sim P_\\text{data}} \\log\\frac{1}{1+\\exp(E_\\theta(x^+))} + \\mathbb{E}_{x^- \\sim P_\\text{LM}} \\log\\frac{1}{1+\\exp(-E_\\theta(x^-))}\n$$\nHowever, the partition function is intractable in practice. The paper proposed a simple way to first sample from the original LM and then to resample from them according to the energy function. This is unfortunately quite expensive.\nTop k samples from the base LM are resampled according to the residual energy function. (Image source:\nDeng et al., 2020\n)\nSmart Prompt Design\n#\nLarge language models have been shown to be very powerful on many NLP tasks, even with only\nprompting\nand no task-specific fine-tuning (\nGPT2\n,\nGPT3\n. The prompt design has a big impact on the performance on downstream tasks and often requires time-consuming manual crafting. For example, factual questions can gain a big boost with smart prompt design in “closed-book exam” (\nShin et al., 2020\n,\nJiang et al., 2020)\n). I’m expecting to see an increasing amount of literature on automatic smart prompt design.\nGradient-based Search\n#\nAutoPrompt\n(\nShin et al., 2020\n;\ncode\n) is a method to automatically create prompts for various tasks via gradient-based search. AutoPrompt constructs a prompt by combining the original task inputs $x$ with a collection of trigger tokens $x_\\text{trig}$ according to a template $\\lambda$. The trigger tokens are shared across all inputs and thus\nuniversally\neffective.\nThe overview of AutoPrompt. The trigger tokens are retrieved to optimize for the target outputs across all inputs. (Image source:\nShin et al., 2020\n)\nThe universal trigger tokens are identified using a gradient-guided search strategy same as in\nWallace et al., 2019\n. The\nuniversal\nsetting means that the trigger tokens $x_\\text{trig}$ can optimize for the target output $\\tilde{y}$ for all inputs from a dataset:\n$$\nx_\\text{trig} = \\arg\\min_{x’_\\text{trig}} \\mathbb{E}_{x\\sim\\mathcal{X}} [\\mathcal{L}(\\tilde{y}, f(x’_\\text{trig}; x))]\n$$\nThe search operates in the embedding space. The embedding of every trigger token  $e_{\\text{trig}_i}$ is first initialized to some default value and then gets updated to minimize the first-order Taylor expansion of the task-specific loss around the current token embedding:\n$$\ne^{(t+1)}_\\text{trig} = \\arg\\min_{e\\in\\mathcal{V}} [e - e^{(t)}_{\\text{trig}_i}]^\\top \\nabla_{e^{(t)}_{\\text{trig}_i}} \\mathcal{L}\n$$\nwhere $\\mathcal{V}$ refers to the embedding matrix of all the tokens. $\\nabla_{e^{(t)}_{\\text{trig}_i}} \\mathcal{L}$ is the average gradient of the task loss over a batch at iteration $t$. We can brute-force the optimal $e$ by a $\\vert \\mathcal{V} \\vert d$-dimensional dot product, which is cheap and can be computed in parallel.\nWe search for trigger tokens by updating their embeddings with the gradient of the task loss per batch. (Image source:\nWallace et al., 2019\n)\nThe above token replacement method can be augmented with beam search. When looking for the optimal token embedding $e$, we can pick top-$k$ candidates instead of a single one, searching from left to right and score each beam by $\\mathcal{L}$ on the current data batch.\nExample prompts discovered by AutoPrompt for different tasks. (Image source:\nShin et al., 2020\n)\nSmart prompt design essentially produces efficient context that can lead to desired completion. Motivated by this observation,\nLi & Liang (2021)\nproposed\nPrefix-Tuning\nwhich assigns a small number of trainable parameters at the beginning of an input sequence (named “prefix”) to steer a LM, $[\\text{PREFIX}; x; y]$. Let $\\mathcal{P}_\\text{idx}$ be a set of prefix indices and $\\text{dim}(h_i)$ be the embedding size. The prefix parameters $P_\\theta$ has the dimension $\\vert\\mathcal{P}_\\text{idx}\\vert \\times \\text{dim}(h_i) $ and the hidden state takes the form:\n$$\nh_i = \\begin{cases}\nP_\\theta[i,:], & \\text{if }i \\in \\mathcal{P}_\\text{idx}\\\\\n\\text{LM}_\\phi(z_i, h_{<{i}}), & \\text{otherwise}\n\\end{cases}\n$$\nNote that only $P_\\theta$ is trainable and the LM parameters $\\phi$ is frozen during training.\nIllustrations of fine-tuning versus prefix-tuning. (Image source:\nLi & Liang 2021\n)\nThe prefix parameters do not tie to any embeddings associated with the real words and thus they are more\nexpressive\nfor steering the context. Direct optimizing $P_\\theta$ unfortunately results in poor performance. To reduce the difficulty associated with high dimensionality training, the matrix $P_\\theta$ is reparameterized by a smaller matrix $P’_\\theta \\in \\mathbb{R}^{\\vert\\mathcal{P}_\\text{idx}\\vert \\times c}$ and a large feed forward network $\\text{MLP}_\\theta \\in \\mathbb{R}^{c\\times \\text{dim}(h_i)}$.\nThe performance increases with the prefix length $\\vert\\mathcal{P}_\\text{idx}\\vert$ up to some value. And this value varies with tasks.\nTask performance, summarization (left) and table-to-text (right), as a function of prefix length. (Image source:\nLi & Liang 2021\n)\nA few other interesting learnings from their ablation studies include:\nTuning only the embedding layer (without prefix) is not sufficiently expressive.\nPlacing the trainable parameter between $x$ and $y$, $[x; \\text{INFIX}; y]$, slightly underperforms prefix-tuning, likely because it only affects the context for $y$ while prefix affects both.\nRandom initialization of $P_\\theta$ leads to low performance with high variance. In contrast, initializing $P_\\theta$ with activations of real words improves generation, even the words are irrelevant to the task.\nFine-tuned models achieve better task performance but they can fail in the low data regime. Both AutoPrompt and Prefix-Tuning were found to outperform fine-tuning in the regime where the training dataset is small (i.e. $10^2-10^3$ samples). As an alternative to fine-tuning, prompt design or learning the context embedding is much cheaper. AutoPrompt improves the accuracy for sentiment classification a lot more than manual prompts and achieves similar performance as linear probing. For the NLI task, AutoPrompt obtains higher accuracy than linear probing. It is able to retrieve facts more accurately than manual prompts too. In low data regime, Prefix-Tuning achieves performance comparable with fine-tuning on table-to-text generation and summarization.\nTwo successive works,\nP-tuning\n(\nLiu et al. 2021\n;\ncode\n) and\nPrompt Tuning\n(\nLester et al. 2021\n), follow the similar idea of explicit training continuous prompt embeddings but with a few different choices over the trainable parameters and architecture. Different from Prefix-Tuning which concatenates continuous prompt tokens in every hidden state layer of the transformer, both P-tuning and Prompt Tuning non-invasively add continuous prompts\nonly in the input\nto work well.\nLet $[P_i]$ be the $i$-th token in the prompt template of\nP-tuning\n(\nLiu et al. 2021\n), we can denote a prompt as a sequence $T=\\{[P_{0:i}], \\mathbf{x}, [P_{i+1:m}], \\mathbf{y}\\}$. Each token $[P_i]$ does not have to be a real token in the model vocabulary (“pseudo-token”), and thus the encoded template $T^e$ looks like the following and the pseudo-token hidden state can be optimized with gradient descent.\n$$\nT^e = \\{ h_0, \\dots, h_i, \\text{embed}(\\mathbf{x}), h_{i+1}, \\dots, h_m, \\text{embed}(\\mathbf{y})\\}\n$$\nThe illustration of P-tuning. Sometimes, adding a few task-related anchor tokens, such as “capital” in the figure, can bring further improvement. (Image source:\nLiu et al. 2021\n)\nThere are two major optimization challenges in P-tuning:\nDiscreteness: The word embedding of a pretrained language model are highly discrete. It is hard to optimize $h_i$ if they are intialized at random.\nAssociation: $h_i$ should be dependent on each other. Thus they develop a mechanism to model this dependency by training a light-weighted LSTM-based prompt encoder:\n$$\nh_i = \\text{MLP}([\\text{LSTM}(h_{0:i}): \\text{LSTM}(h_{i:m})])\n$$\nP-tuning is more flexible than prefix-tuning, as it inserts trainable tokens in the middle of a prompt not just at the beginning. The usage of task-specific anchor tokens is like combining manual prompt engineering with trainable prompts.\nPrompt Tuning\n(\nLester et al. 2021\n) largely simplifies the idea of prefix tuning by only allowing an additional $k$ tunable tokens per downstream task to be prepended to the input text. The conditional generation is $p_{\\theta, \\theta_P}(Y \\vert [P; X])$, where $P$ is the “pseudo prompt” with parameters $\\theta_P$ trainable via back-propagation. Both $X$ and $P$ are embedding vectors and we have $X \\in \\mathbb{R}^{n \\times d^e}, P \\in \\mathbb{R}^{k \\times d^e}$ and $[P;X] \\in \\mathbb{R}^{(n+k) \\times d^e}$, where $d^e$ is the embedding space dimensionality.\nPrompt tuning produces competitive results as model fine-tuning when the model gets\nlarge\n(billions of parameters and up). This result is especially interesting given that large models are expensive to fine-tune and execute at inference time.\nWith learned task-specific parameters, prompt tuning achieves better transfer learning when adapting to new domains. It outperforms fine-tuning on domain shift problems.\nThey also showed that prompt ensembling of multiple prompts for the same task introduces further improvement.\nThe illustration of how Prompt Tuning works. (Image source:\nLester et al. 2021\n)\nThe experiments investigated several prompt initialization schemes:\nRandom initialization by uniformly sampling from [-0.5, 0.5];\nSample embeddings of top 5000 common tokens;\nUse the embedding values of the class label strings. If we don’t have enough class labels to initialize the soft-prompt, we fall back to scheme 2.\nRandom initialization performs noticeably worse than the other two options.\nThe effect of (a) different prompt initialization schemes and (b) different prompt lengths. (Image source:\nLester et al. 2021\n)\nThe pre-training objectives also have a big impact on the quality of prompt tuning. T5’s “span corruption” is not a good option here.\nPrompt tuning is found to be less likely to overfit to a specific dataset. To evaluate the robustness to data shifting problem, they trained the model on one dataset of one task and evaluated it on the test dataset but in a\ndifferent domain\n. Prompt tuning is more resilient and can generalize to different domains better.\nPrompt tuning is more resilient to domain shift between train and test sets. (Image source:\nLester et al. 2021\n)\nHeuristic-based Search\n#\nParaphrasing is a quick way to explore more prompts similar to the known version, which can be done via\nback-translation\n.  Using back-translation, the initial prompt is translated into $B$ candidates in another language and then each is translated back into $B$ candidates in the original language. The resulting total $B^2$ candidates are scored and ranked by their round-trip probabilities.\nRibeiro et al (2018)\nidentified\nsemantically equivalent adversaries (SEA)\nby generating a variety of paraphrases $\\{x’\\}$ of input $x$ until it triggers a different prediction of target function $f$:\n$$\n\\begin{aligned}\nSEA(x, x') &= \\mathbb{1}[\\text{SemEq}(x, x') \\land f(x) \\neq f(x')] \\\\\n\\text{where SemEq}(x, x') &= \\mathbb{1}[\\min\\Big(1, \\frac{p(x'\\vert x)}{p(x\\vert x)} \\Big) \\geq \\tau]\n\\end{aligned}\n$$\nwhere the score $p(x’\\vert x)$ is proportional to translating $x$ into multiple languages and then translating it back to the original language.\nExamples of SEA rules include (\nWhat\nNOUN\n→Which\nNOUN\n), (\nWP\nis →\nWP\n’s’\n), (\nwas→is\n), etc. They are considered as “bugs” in the model. Applying those rules as data augmentation in model training helps robustify the model and fix bugs.\nJiang et al (2020)\nattempts to validate whether a trained language model knows certain knowledge by automatically discovering better prompts to query. Within the scope of knowledge retrieval where factual knowledge is represented in the form of a triple $\\langle x, r, y \\rangle$ (subject, relation, object). The prompts can be mined from training sentences (e.g. Wikipedia description) or expanded by paraphrase.\nInterestingly some small modifications in the prompts may lead to big gain, as shown in Fig. X.\nSmall modifications in prompt templates can lead to big performance gains: replacement in blue, insertion in green, deletion in red. (Image source:\nJiang et al., 2020\n)\nFine-tuning\n#\nFine-tuning is an intuitive way to guide a LM to output desired content, commonly by training on supervised datasets or by RL. We can fine-tune all the weights in the model or restrict the fine-tuning to only top or additional layers.\nConditional Training\n#\nConditional training aims to learn a generative model conditioned on a control variable $z$, $p(y \\vert x, z)$.\nFan et al (2018)\ntrained a conditional language model for 2-step story generation. First, a model outputs the story sketch and then a story writing model creates a story following that sketch. The mechanism of conditioning on the sketch is implemented by a\nfusion\nmodel architecture. The fusion model enforces a form of\nresidual learning\nthat allows the story writing model to focus on learning what the first sketch generation model is missing. Also for story generation,\nPeng et al (2018)\nexperimented with an ending valence-conditioned story generator LM, $p(x_t \\vert x_{<t}, z)$ where $z$ is the label of the story ending (sad, happy or neutral). Their language model is a bidirectional LSTM and the label is mapped into a learned embedding which then blends into the LSTM cell.\nCTRL\n(\nKeskar et al., 2019\n;\ncode\n) aims to train a language model conditioned control code $z$ using controllable datasets. CTRL learns the conditioned distribution $p(x \\vert z)$ by training on raw text sequences with\ncontrol code prefixes\n, such as\n[horror]\n,\n[legal]\n, etc. Then the learned model is able to generate text with respect to the prompt prefix. The training data contains Wikipedia, OpenWebText, books, Amazon reviews, reddit corpus and many more, where each dataset is assigned with a control code and subreddit in the reddit corpus has its own topic as control code.\nDatasets used for training CTRL and associated control codes. (Image source: Edited from Table 7 in\nKeskar et al., 2019\n)\nThe control code also can be used for\ndomain annotation\ngiven tokens, because $p(z \\vert x) \\propto p(x \\vert z) p(z)$, assuming the prior over domains is uniform. One limitation of CTRL is the lack of control for\nwhat not to generate\n(e.g. avoid toxicity).\nThe examples of conditioned sample generation by CTRL. (Image source:\nKeskar et al., 2019\n)\nNote that CTRL trains a transformer model from scratch. However, labelling all the text within the same dataset with the same control code (e.g. All the wikipedia articles have “wikipedia” as control code) feels quite constrained. Considering that often we need highly customized control codes but only have a limited amount of labelled data, I would expect fine-tuning an unconditional LM with a small labelled dataset in the same way as CTRL to work out well too. Although how much data is needed and how good the sample quality might be are subject to experimentation.\nRL Fine-tuning\n#\nFine-tuning a sequential model with RL regarding any arbitrary and possibly non-differentiable reward function has been proved to work well years ago (\nRanzato et al., 2015\n). RL fine-tuning can resolve several problems with\nteacher forcing\nmethod. With teacher forcing, the model only minimizes a maximum-likelihood loss at each individual decoding step during training but it is asked to predict the entire sequence from scratch at test time. Such a discrepancy between train and test could lead to exposure bias and accumulated error. In contrast, RL fine-tuning is able to directly optimize task-specific metrics on the sequence level, such as BLEU for translation (\nRanzato et al., 2015\n,\nWu et al., 2016\n,\nNguyen et al., 2017\n), ROUGE for summarization (\nRanzato et al., 2015\n,\nPaulus et al., 2017\n,\nWu and Hu, 2018\n) and customized metric for story generation (\nTambwekar et al., 2018\n).\nRanzato et al (2015)\napplied REINFORCE to train RNN models for sequence generation tasks. The model is first trained to predict the next token using cross-entropy loss (ML loss) and then fine-tuned alternatively by both ML loss and REINFORCE (RL loss). At the second fine-tuning stage, the number of training steps for next-token prediction is gradually decreasing until none and eventually only RL loss is used. This sequence-level RL fine-tuning was shown by experiments to lead to great improvements over several supervised learning baselines back then.\nGoogle implemented the similar approach in their neural machine translation system (\nWu et al., 2016\n) and\nPaulus et al (2017)\nadopted such approach for summarization task. The training objective contains two parts, ML loss for next token prediction, $\\mathcal{L}_\\text{ML} = \\sum_{(x, y^*)\\sim\\mathcal{D}} \\log p_\\theta(y^* \\vert x)$, and RL loss $\\mathcal{L}_\\text{RL}$ for maximizing the expected reward where the reward per sequence is measured by BLEU or ROUGE. The model is first trained with $\\mathcal{L}_\\text{ML}$ until convergence and then fine-tuned with a linear combination of two losses, $\\mathcal{L}_\\text{mix} = \\alpha \\mathcal{L}_\\text{ML} + (1 - \\alpha)\\mathcal{L}_\\text{RL}$.\nThe RL loss of Google NMT is to maximize the expected BLEU score:\n$$\n\\mathcal{L}_\\text{RL} = - \\sum_{(x, y^*)\\sim\\mathcal{D}} \\mathbb{E}_{y\\sim p_\\theta(.\\vert x)} [R(y, y^*)]\n$$\nwhere $y$ is the predicted sequence and $y^*$ is the ground truth.\nPaulus et al (2017)\nadded an extra weighting term based on the reward difference between two output sequences, $y$ by sampling the next token according to the predicted probability and $\\hat{y}$ by greedily taking the most likely token. This RL loss maximizes the conditional likelihood of the sampled sequence $y$ if it obtains a higher reward than the greedy baseline $\\hat{y}$:\n$$\n\\mathcal{L}_\\text{RL} = \\sum_{(x, y^*)\\sim\\mathcal{D}} (R(\\hat{y}, y^*) - R(y, y^*)) \\sum_{t=1}^{n'} \\log p(y_t \\vert y_{<{t}}, x)\n$$\nRL Fine-tuning with Human Preferences\n#\nReward learning is critical for defining human preferences. Quantitative measurement like BLEU or ROUGE computes the overlap of words and n-gram phrases between sequences and does not always correlate with better quality by human judges. Reward learning from human feedback (\nChristiano et al., 2017\n) is a better way to align what we measure with what we actually care about. Human feedback has been applied to learn a reward function for applications like story generation (\nYi et al., 2019\n) and summarization (\nBöhm et al., 2019\n,\nZiegler et al., 2019\n,\nStiennon et al., 2020\n).\nIn order to generate more coherent conversation,\nYi et al (2019)\ncollected 4 types of binary human feedback given a conversation pair (user utterance, system response), whether the system response is (1) comprehensive, (2) on topic, (3) interesting and (4) leading to continuation of the conversation.\nAn evaluator is trained to predict human feedback and then is used to rerank the beam search samples, to finetune the model or to do both. (Actually they didn’t use RL fine-tuning but rather use the evaluator to provide a discriminator loss in supervised fine-tuning.)\nLet’s define a learned reward function $R_\\psi(x, y)$ parameterized by $\\psi$ as a measurement for the quality of output $y$ given the input $x$.\nTo learn the ground truth reward $R^*$ defined by human judgements,\nBöhm et al (2019)\ncompared two loss functions:\n(1) Regression loss: simply minimizing the mean squared error.\n$$\n\\mathcal{L}^\\text{MSE}_\\text{rm} = [R^*(x, y) - R_\\psi(x, y)]^2\n$$\n(2) Preference loss: learning to agree with the ground truth reward,\n$$\n\\begin{aligned}\n\\mathcal{L}^\\text{pref}_\\text{rm} =& - \\sum_{i,j} \\big(\\mathbb{1}[R^*(x, y_i) > R^*(x, y_j)] \\log P(y_i \\succ y_j) + \\\\\n&\\mathbb{1}[R^*(x, y_j) > R^*(x, y_i)] \\log P(y_j \\succ y_i) \\big)\\\\ \n\\text{where }P(y_i \\succ y_j) =& \\frac{\\exp(R_\\psi(x, y_i))}{\\exp(R_\\psi(x, y_i)) + \\exp(R_\\psi(x, y_j))}\n\\end{aligned}\n$$\nTheir experiments showed that the\npreference loss\nachieves the best performance, where the reward model is a thin MLP layer on top of BERT sentence embedding.\nZiegler et al (2019)\ncollected human labels by asking humans to select the best candidate $y_b$ out of a few options $\\{y_i\\}$ given the input $x \\sim \\mathcal{D}$. The candidates are sampled by $y_0, y_1 \\sim p(.\\vert x), y_2, y_3 \\sim \\pi(.\\vert x)$. We should be aware that human labeling might have very high disagreement when the ground truth is fuzzy.\nThe overview of the training framework for fine-tuning a language model policy with reward learned from human feedback. (Image source:\nZiegler et al., 2019\n)\nThe reward model is implemented by a pretrained language model with an extra random linear layer of the final embedding output. It it trained to minimize the loss:\n$$\n\\mathcal{L}_\\text{rm} = -\\mathbb{E}_{(x, \\{y_i\\}, b) \\sim \\mathcal{D}} \\Big[ \\log \\frac{\\exp(R_\\psi(x, y_b))}{\\sum_i \\exp(R_\\psi(x, y_i))} \\Big]\n$$\nTo keep the scale consistent during training, the reward model is normalized to have mean 0 and variance 1.\nDuring RL fine-tuning, the policy $\\pi$, initialized by a pretrained language model $p$, is optimized via\nPPO\nwith the above learned reward model. To avoid the policy’s deviating from its original behavior too much, a\nKL penalty\nis added:\n$$\nR(x, y) = R_\\psi(x, y) - \\beta\\log\\frac{\\pi(y \\vert x)}{p(y \\vert x)}\n$$\nIf running online data collection, human label collection process is continued during RL fine-tuning and thus the human labelers can review results generated by the latest policy. The number of human labels are evenly spread out during the training process. Meanwhile the reward model is also retrained periodically. Online data collection turns out to be important for the summarization task but not for the text continuation task. In their experiments, jointly training the reward model and the policy with shared parameters did not work well and can lead to overfitting due to the big imbalance between dataset sizes.\nIn the following work (\nStiennon et al., 2020\n), the human label collection was further simplified to select the best option between a pair of summaries, $y_b \\in\\{y_0, y_1\\}$ The reward model loss was updated to optimize the log odds of the selected summary:\n$$\n\\mathcal{L}_\\text{rm} = - \\mathbb{E}_{(x, y_0, y_1, b)\\sim\\mathcal{D}} [\\log(\\sigma(r_\\theta(x, y_b) − r_\\theta(x, y_{1−b})))]\n$$\nThe overview of fine-tuning the language model policy from human feedback for summarization, including (1) human feedback collection, (2) reward model training, and (3) policy training. (Image source:\nStiennon et al., 2020\n)\nGuided Fine-tuning with Steerable Layer\n#\nInstead of fine-tuning the entire model, only fine-tuning a small extra set of parameters while the base model stays fixed is computationally cheaper.\nIn computer vision, plug-and-play generative networks (PPGN;\nNguyen et al., 2017\n) generate images with different attributes by plugging a discriminator $p(a \\vert x)$ into a base generative model $p(x)$. Then the sample with a desired attribute $a$ can be sampled from $p(x \\vert a) \\propto p(a \\vert x)p(x)$. Inspired by PPGN, the\nplug-and-play language model\n(\nPPLM\n;\nDathathri et al., 2019\n) combines one or multiple simple attribute models with a pretrained language model for controllable text generation.\nGiven an attribute $a$ and the generated sample $x$, let an attribute model be $p(a\\vert x)$. To control content generation, the current latent representation at time $t$, $H_t$ (containing a list of key-value pairs per layer), can be shifted by $\\Delta H_t$  in the direction of the sum of two gradients:\nOne toward higher log-likelihood of the attribute $a$ under $p(a \\vert x)$ — so that the output content acquires a desired attribute.\nThe other toward higher log-likelihood of the unmodified language model $p(x)$ — so that the generated text is still in fluent and smooth natural language.\nTo shift the output, at decoding time, PPLM runs one forward → one backward → one forward, three passes in total:\nFirst a forward pass is performed to compute the likelihood of attribute $a$ by $p(a\\vert x)$;\nLet $\\Delta H_t$ be a stepwise update to the hidden state $H_t$ such that $(H_t + \\Delta H_t)$ shifts the distribution of generated text closer to having the attribute $a$. $\\Delta H_t$ is initialized at zero.\nThen a backward pass updates the LM hidden states using normalized gradients from the attribute model $\\nabla_{\\Delta H_t} \\log p(a \\vert H_t + \\Delta H_t)$ as\n$$\n\\Delta H_t \\leftarrow \\Delta H_t + \\alpha \\frac{\\nabla_{\\Delta H_t} \\log p(a|H_t + \\Delta H_t)}{\\| \\nabla_{\\Delta H_t} \\log p(a|H_t + \\Delta H_t) \\|^\\gamma}\n$$\nwhere $\\gamma$ is a normalization scaling coefficient, set per layer. $\\alpha$ is step size. This update can be repeated $m \\in [3, 10]$ times\n3. The final forward pass recomputes a new distribution over the vocabulary, generated from the updated latents $\\tilde{H}_t = H_t + \\Delta H_t$. The next token is sampled from the updated distribution.\nThe overview of how PPLM runs three passes to update the model output to increase the likelihood of a desired attribute. (Image source:\nDathathri et al., 2019\n)\nMultiple attribute models can be mix-and-matched during generation with customized weights, acting as a set of “control knobs”. The PPLM paper explored two types of attribute models:\nThe simplest attribution model is based on a predefined\nbag of words\n(BoW), $\\{w_1, \\dots, w_k\\}$, that specifies a topic of interest.\n$$\n\\log p(a \\vert x) = \\log\\big( \\sum_{i=1}^k p_{t+1} [w_i] \\big)\n$$\nTo encourage the model to output the desired words at least once but not at every step, they normalize the gradient by the maximum gradient norm.\nInterestingly, they found that increasing the probability of generating words in the bag also increases the probability of generating\nrelated\nbut not identical words about the same topic.\n2. The discriminator attribute models are based on learned classifiers which define preferences by a distribution instead of hard samples.\nTo ensure the fluency in language, PPLM applied two additional designs:\nMinimizing the KL diverge between modified and unmodified LM, commonly seen in other RL fine-tuning approaches (see\nabove\n).\nIt performs\npost-norm fusion\nto constantly tie the generated text to the unconditional LM $p(x)$, $x_{t+1} \\sim \\frac{1}{\\beta}(\\tilde{p}_{t+1}^{\\gamma_\\text{gm}} p_{t+1}^{1-\\gamma_\\text{gm}})$, where $p_{t+1}$ and $\\tilde{p}_{t+1}$ are the unmodified and modified output distributions, respectively. $\\beta$ is a normalizing factor. $\\gamma_\\text{gm} \\in [0.8, 0.95]$ balances between prediction from before and after models.\nExamples of controllable text generation by PPLM. (Image source:\nDathathri et al., 2019\n)\nInterestingly, they found a large variance in the extent of controllability across topics. Some topics (religion, science, politics) are easier to control for compared to others (computers, space).\nOne obvious drawback of PPLM is that due to multiple passes at every decoding step, the test time computation becomes much more expensive.\nSimilar to PPLM,\nDELOREAN\n(DEcoding for nonmonotonic LOgical REAsoNing;\nQin et al., 2020\n) incorporates the future context by back-propagation. Given input text $\\mathbf{x}$, DELOREAN aims to generate continuation completion $\\mathbf{y} = [y_1, \\dots, y_N]$ such that $y$ satisfies certain constraints defined by a context $z$. To keep the generation differentiable, a soft representation of $y$ is tracked, $\\tilde{\\mathbf{y}}=(\\tilde{y}_1, \\dots, \\tilde{y}_N)$ where $\\tilde{y}_i \\in \\mathbb{R}^V$ are logits over the vocabulary. $\\tilde{\\mathbf{y}}^{(t)}$ is the soft representation at iteration $t$.\nGiven the representation $\\tilde{y}^{(t-1)}$ at iteration $t$, it runs the following procedures:\nBackward\n: The constraint is represented as a loss function $\\mathcal{L}(\\mathbf{x}, \\tilde{\\mathbf{y}}^{(t-1)}, z))$. The logits are updated via gradient descent: $\\tilde{y}^{(t), b}_n = \\tilde{y}_n^{(t-1)} - \\lambda \\nabla_{\\tilde{y}_n} \\mathcal{L}(\\mathbf{x}, \\tilde{\\mathbf{y}}^{(t-1)}, z)$.\nForward\n: Run forward pass to ensure the generated text is fluent. $\\tilde{y}^{(t),f}_n = \\text{LM}(\\mathbf{x}, \\tilde{\\mathbf{y}}^{(t)}_{1:n-1})$.\nThen linearly combine two logits together to create a new representation $\\tilde{y}^{(t)}_n = \\gamma \\tilde{y}^{(t), f}_n + (1-\\gamma) \\tilde{y}^{(t), b}_n$. Note that each $\\tilde{y}^{(t)}_n$ is needed to sample the next $\\tilde{y}^{(t),f}_{n+1}$.\nSide-tuning\n(\nZhang et al., 2019\n) trains a light-weighted side network that learns a residual on top of the original model outputs without modifying the pre-trained model weights. Unlike PPLM, no gradient update is applied on the hidden states. It is a simple yet effective approach for incremental learning. The base model is treated as a black-box model and does not necessarily have to be a neural network. Side-tuning setup assumes the base and side models are fed exactly the same input and the side model is independently learned.\nComparison of fixed weights, fine-tuning and side-tuning. (Image source:\nZhang et al., 2019\n)\nThe paper explored different strategies of fusing predictions from the base and side models:\nproduct\nis the worst while\nsum\n($\\alpha$-blending), MLP, and\nFiLM\nare comparable. Side-tuning is able to achieve better performance, when it is trained with intermediate amounts of data and when the base network is large.\nAuxiliary tuning\n(\nZeldes et al., 2020\n) supplements the original pre-trained model with an\nauxiliary\nmodel that shifts the output distribution according to the target task. The base and auxiliary model outputs are merged on the logits level. The combined model is trained to maximize the likelihood $p(x_t\\vert x_{<t}, z)$ of target output.\nThe conditional probability of $p(x_t\\vert x_{<t}, z)$ can be decomposed into two parts:\n$p(x_t\\vert x_{<t})$ assigns high probabilities to fluent sequences of tokens;\na shift on $p(x_t\\vert x_{<t})$ towards $p(x_t\\vert x_{<t}, z)$.\n$$\np(x_t\\vert x_{<{t}}, z) = \\text{softmax}(\\text{logits}_\\text{LM}(x_t \\vert x_{<{t}}) + \\text{logits}_\\text{aux}(x_t \\vert x_{<{t}}, z))\n$$\nBy Bayesian rule, we have\n$$\np(x_t\\vert x_{<{t}}, z)\n= \\frac{p(z \\vert x_{\\leq t})}{p(z)} p(x_t \\vert x_{<{t}}) \n\\propto p(z \\vert x_{\\leq t}) p(x_t \\vert x_{<{t}})\n$$\nAnd therefore the auxiliary model $\\text{logits}_\\text{aux}(x_t \\vert x_{<t}, z))$ effectively should learn to predict $p(z \\vert x_{\\leq t})$. In the experiments of\nZeldes et al., 2020\n, the auxiliary model can re-use the intermediate layers of the pre-trained LM for feature extraction.\nThe auxiliary model is trained by reusing features extracted from multiple layers of the base model. (Image source:\nZeldes et al., 2020\n)\nGeDi\n(\nKruse et al., 2020\n) guides the text generation by\nGenerative Discriminator\n. The discriminator is implemented as a class conditional language model (CC-LM), $p_\\theta(x_{1:t} \\vert z)$. The discriminator guides generation at each decoding step by computing classification probabilities for all possible next tokens via Bayes rule by normalizing over\ntwo\ncontrastive class-conditional distributions:\nOne conditioned on the control code $z$ for desired attribute.\nThe other conditioned on the anti-control code $\\bar{z}$ for undesired attributes.\nGeDi relies on the contract between $p_\\theta(x_{1:t} \\vert z)$ and $p_\\theta(x_{1:t} \\vert \\bar{z})$ to compute the probability of the sequence belonging to the desired class. The discriminator loss is to maximize the probability of desired attribute $z$:\n$$\n\\begin{aligned}\np_\\theta(z \\vert x_{1:t}) &= \\frac{p(z) p_\\theta(x_{1:\\tau} \\vert z)^{\\alpha/\\tau}}{\\sum_{z' \\in \\{z, \\bar{z}\\}} p(z') p_\\theta(x_{1:\\tau} \\vert z')^{\\alpha/\\tau} } \\\\\n\\mathcal{L}_\\text{desc} \n&= -\\frac{1}{N} \\sum_{i=1}^N \\log p_\\theta(z^{(i)} \\vert x^{(i)}_{1:\\tau_i}) \\\\\n&= -\\frac{1}{N} \\sum_{i=1}^N \\log \\frac{p(z) p_\\theta(x^{(i)}_{1:\\tau_i} \\vert z^{(i)})^{\\alpha/t_i}}{\\sum_{z' \\in \\{z, \\bar{z}\\} } p(z')p_\\theta(x^{(i)}_{1:\\tau_i} \\vert z')^{\\alpha/\\tau_i}}\n\\end{aligned}\n$$\nwhere $p(z) = \\exp(b_z) / \\sum_{z’} \\exp(b_{z’})$ and $b_z$ is a learned class prior. The probabilities are normalized by the current sequence length $\\tau$ to robustify generation sequences of variable lengths. $\\tau_i$ is the sequence length of the $i$-th input $x^{(i)}$ in the dataset.\nAn illustration of how GeDi works via Bayesian rule. (Image source:\nKruse et al., 2020\n)\nThey finetuned a GPT2-medium model with control code similar to how\nCTRL\nis trained to form a CC-LM using a linear combination of discriminative loss and generative loss. This discriminator model is then used as GiDe to guide generation by a larger language model like GPT2-XL.\nOne way of decoding from GeDi is to sample from a weighted posterior $p^w(x_{t+1}\\vert x_{1:t}, z) \\propto p(z \\vert x_{1:t+1})^w p(x_{t+1} \\vert x_{1:t})$ where $w>1$ applies additional bias toward the desired class $z$. In the sampling process, only tokens with the class or next-token probability larger than a certain threshold are selected.\nGeDi guided generation in their experiments showed strong controllability and ran 30x faster than\nPPLM\n.\nDistributional Approach\n#\nGeneration with Distributional Control\n(GDC;\nKhalifa, et al. 2020\n) frames controlled text generation as the optimization of a probability distribution with a constraint. It involves two major steps.\nStep 1: Learn a EBM of the target model\nLet’s label a pretrained LM as $a$ and a target LM with desired features as $p$. The desired features can be defined by a set of pre-defined real-valued feature functions $\\phi_i(x), i=1,\\dots,k$ over $x \\in X$, denoted as a vector $\\boldsymbol{\\phi}$. When sequences $x \\in X$ are sampled according to the desired model $p$, the expectations of features $\\mathbb{E}_{x\\sim p}\\boldsymbol{\\phi}(x)$ should be close to $\\bar{\\boldsymbol{\\mu}}$ , named “\nmoment constraints\n”. The feature function $\\phi_i$ can have distinct values (e.g. identity function for binary classifier) or continuous probabilities. In the meantime, the fine-tuned model $p$ should not diverge from $a$ too much by maintaining a small KL divergence measure.\nIn summary, given a pretrained model $a$, we would like to find a target model $p$ such that:\n$$\n\\begin{aligned}\n\\bar{\\boldsymbol{\\mu}} &= \\mathbb{E}_{x\\sim p}\\boldsymbol{\\phi}(x) \\\\\np &= \\arg\\min_{c \\in \\mathcal{C}} D_\\text{KL}(c, a)\n\\end{aligned}\n$$\nwhere $\\mathcal{C}$ is the set of all distributions over $X$ that satisfy the moment constraints.\nAccording to theorems in Information Geometry, $p$ can be approximated by an EBM (energy-based model; an unnormalized probability distribution) $P$ in the form of exponential function, such that $p(x) \\propto P(x)$ and $p(x)=\\frac{1}{Z}P(x)$ where $Z=\\sum_x P(x)$. The energy-based model can be approximated by:\n$$\nP(x)=a(x)\\exp\\big(\\sum_i \\lambda_i \\phi_i(x)\\big)=a(x)\\exp(\\boldsymbol{\\lambda}\\cdot\\boldsymbol{\\phi}(x))\n$$\nLet’s define\nimportance weight\n$w(x, \\boldsymbol{\\lambda}) = \\frac{P(x)}{a(x)} = \\exp\\langle\\boldsymbol{\\lambda}\\cdot\\boldsymbol{\\phi}(x)\\rangle$. Given a large number of sequences sampled from the pretrained model $x_1, \\dots, x_N \\sim a(x)$,\n$$\n\\begin{aligned}\n\\mu(\\boldsymbol{\\lambda}) \n&= \\mathbb{E}_{x\\sim p}\\boldsymbol{\\phi}(x)\n= \\mathbb{E}_{x\\sim a} \\frac{p(x)}{a(x)}\\boldsymbol{\\phi}(x)\n= \\frac{1}{Z}\\mathbb{E}_{x\\sim a} w(x, \\boldsymbol{\\lambda}) \\boldsymbol{\\phi}(x) \\\\\n&= \\frac{\\mathbb{E}_{x\\sim a} w(x, \\boldsymbol{\\lambda}) \\boldsymbol{\\phi}(x)}{\\sum_{x\\in X} P(x)}\n= \\frac{\\mathbb{E}_{x\\sim a} w(x, \\boldsymbol{\\lambda}) \\boldsymbol{\\phi}(x)}{\\sum_{x\\in X} w(x, \\boldsymbol{\\lambda})a(x)}\n= \\frac{\\mathbb{E}_{x\\sim a} w(x, \\boldsymbol{\\lambda}) \\boldsymbol{\\phi}(x)}{\\mathbb{E}_{x\\sim a} w(x, \\boldsymbol{\\lambda})} \\\\\n&\\simeq \\frac{\\sum_{i=1}^N w(x_i,\\boldsymbol{\\lambda}) \\boldsymbol{\\phi}(x_i)}{\\sum_{i=1}^N w(x_i, \\boldsymbol{\\lambda})}\n= \\frac{\\sum_{i=1}^N \\exp\\langle\\boldsymbol{\\lambda}\\cdot\\boldsymbol{\\phi}(x)\\rangle \\boldsymbol{\\phi}(x_i)}{\\sum_{i=1}^N \\exp\\langle\\boldsymbol{\\lambda}\\cdot\\boldsymbol{\\phi}(x)\\rangle}\n\\end{aligned}\n$$\nUsing SGD over the objective $|\\boldsymbol{\\mu}(\\boldsymbol{\\lambda}) - \\bar{\\boldsymbol{\\mu}}|^2_2$, we can obtain an estimated value for $\\boldsymbol{\\lambda}$ and a representation of $P(x)=a(x)\\exp\\langle\\boldsymbol{\\lambda}\\cdot\\boldsymbol{\\phi}(x)\\rangle$. $P(x)$ is a sequential EBM because $a$ is an autoregressive model.\nStep 2: Learn the target probability distribution\nThe EBM $P(x)$ can compute ratios of probabilities of two sequences, but cannot sample from $p(x)$ with knowing $Z$. In order to sample from a sequential EBM, the paper proposed to use\nDistributional Policy Gradient\n(DPG; but not this\nDPG\n) with the objective to obtain an autoregressive policy $\\pi_\\theta$ to approximate a target distribution $p$ by minimizing the cross entropy $H(p, \\pi_\\theta)$. DPG runs through a sequence of iterations. Within each iteration, the proposed distribution $q$ is used for sampling and we can correct the cross entropy loss with importance weights too:\n$$\n\\begin{aligned}\n\\nabla_\\theta H(p, \\pi_\\theta) \n&= - \\nabla_\\theta \\mathbb{E}_{x\\sim p} \\log \\pi_\\theta(x)\n= - \\mathbb{E}_{x\\sim p} \\nabla_\\theta  \\log \\pi_\\theta(x) \\\\\n&= - \\mathbb{E}_{x\\sim q} \\frac{p(x)}{q(x)} \\nabla_\\theta  \\log \\pi_\\theta(x)\n= - \\frac{1}{Z}\\mathbb{E}_{x\\sim q} \\frac{P(x)}{q(x)} \\nabla_\\theta  \\log \\pi_\\theta(x)\n\\end{aligned}\n$$\nTo learn such a $\\pi_\\theta$, the paper adopts a KL-adaptive version of DPG: It only updates $q$ when the estimated policy $\\pi_\\theta$ gets closer to $p$. This adaptive step is important for fast convergence.\nThe algorithm of distributional policy gradient to make it possible to sample from a EBM $P(x)$, where $q$ is initialized to be $a$. (Image source:\nKhalifa, et al. 2020\n)\nThis approach can be used to model various constraints in controllable text generation:\nPointwise constraints: $\\phi_i$ is a binary feature; such as constraining the presence or absence of words, or classifier-based constraints.\nDistributional constraints: $\\phi_i$ represents a probability distribution; such as constraining the probability of gender, topic, etc. Their experiments showed great progress in debiasing a GPT-2 model that was trained on Wikipedia Biographies corpus. The percentage of generated biographies on females increased from 7.4% to 35.6%.\nHybrid constraints: combine multiple constraints by simply summing them up.\nDebiasing experiments using GDC with various constraints. (Image source:\nKhalifa, et al. 2020\n)\nCompared to other baselines, GDC using pointwise constraints diverges less from the base model $a$ and produces smoother curves.\nCompare pointwise constrained GDC with several baselines. Low Self-BLEU-5 and high Dist-1 indicate high diversity. (Image source:\nKhalifa, et al. 2020\n)\nREINFORCE that optimizes the reward $\\phi$ directly ($\\text{REINFORCE}$ in Fig. X.) without constraints converges fast but has a high deviation from the original model.\nREINFORCE that optimizes $P(x)$ ($\\text{REINFORCE}_{P(x)}$ in Fig. X.) has low sample diversity.\nCompared to\nZiegler et al., 2019\nGDC has smoother learning curves and produces a richer vocabulary.\nUnlikelihood Training\n#\nThe standard way of maximizing the log-likelihood loss in language model training leads to\nincorrect token distribution\n, which cannot be fixed with only smart decoding methods. Such models tend to output high-frequency words too often and low-frequency words too rarely, especially when using deterministic decoding (e.g. greedy, beam search). In other words, they are overconfident in their predictions.\nUnlikelihood training (\nWelleck & Kulikov et al. 2019\n] tries to combat this and incorporates preference to\nunwanted\ncontent into the training objective directly. It combines two updates:\nA routine maximized likelihood update to assign true tokens with high probability;\nA new type of unlikelihood update to avoid unwanted tokens with high probability.\nGiven a sequence of tokens $(x_1, \\dots, x_T)$ and a set of negative candidate tokens $\\mathcal{C}^t = \\{c_1, \\dots , c_m\\}$ at step $t$, where each token $x_i, c_j \\in \\mathcal{V}$, the combined loss for step $t$ is defined as:\n$$\n\\mathcal{L}^t_\\text{UL}(p_\\theta (. \\vert x_{<{t}}), \\mathcal{C}^t)\n= - \\alpha \\cdot \\underbrace{\\sum_{c \\in \\mathcal{C}^t} \\log(1 - p_\\theta(c \\vert x_{<{t}}))}_\\text{unlikelihood} - \\underbrace{\\log p_\\theta (x_t \\vert x_{<{t}})}_\\text{likelihood}\n$$\nOne approach for constructing $\\mathcal{C}^t$ is to randomly select candidates from model-generated sequences.\nThe unlikelihood training can be extended to be on the\nsequence\n-level, where the negative continuation is defined by a sequence of per-step negative candidate sets. They should be designed to penalize properties that we don’t like. For example, we can penalize repeating n-grams as follows:\n$$\n\\mathcal{C}^t_\\text{repeat-n} = \\{x_t\\} \\text{ if }(x_{t-i}, \\dots, x_{t+j}) \\in x_{<{t-i}} \\text{ for any } (j-i)=n, i\\leq n \\leq j.\n$$\nTheir experiments used unlikelihood training to avoid repetitions in language model outputs and indeed showed better results on less repetition and more unique tokens compared to standard MLE training.\nCitation\n#\nCited as:\nWeng, Lilian. (Jan 2021). Controllable neural text generation. Lil’Log. https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/.\nOr\n@article{weng2021conditional,\n  title   = \"Controllable Neural Text Generation.\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2021\",\n  month   = \"Jan\",\n  url     = \"https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/\"\n}\nReferences\n#\n[1] Patrick von Platen.\n“How to generate text: using different decoding methods for language generation with Transformers”\nHugging face blog, March 18, 2020.\n[2] Angela Fan, et al.\n“Hierarchical Neural Story Generation/”\narXiv preprint arXiv:1805.04833 (2018).\n[3] Ari Holtzman et al.\n“The Curious Case of Neural Text Degeneration.”\nICLR 2020.\n[4] Marjan Ghazvininejad et al.\n“Hafez: an interactive poetry generation system.”\nACL 2017.\n[5] Ari Holtzman et al.\n“Learning to write with cooperative discriminators.”\nACL 2018.\n[6] Ashutosh Baheti et al.\n“Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints.”\nEMNLP 2018.\n[7] Jiatao Gu et al.\n“Trainable greedy decoding for neural machine translation.”\nEMNLP 2017.\n[8] Kyunghyun Cho.\n“Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model.”\narXiv preprint arXiv:1605.03835. (2016).\n[9] Marco Tulio Ribeiro et al.\n“Semantically equivalent adversarial rules for debugging NLP models.”\nACL 2018.\n[10] Eric Wallace et al.\n“Universal Adversarial Triggers for Attacking and Analyzing NLP.”\nEMNLP 2019. [\ncode\n]\n[11] Taylor Shin et al.\n“AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.”\nEMNLP 2020. [\ncode\n]\n[12] Zhengbao Jiang et al.\n“How Can We Know What Language Models Know?”\nTACL 2020.\n[13] Nanyun Peng et al.\n“Towards Controllable Story Generation.”\nNAACL 2018.\n[14] Nitish Shirish Keskar, et al.\n“CTRL: A Conditional Transformer Language Model for Controllable Generation”\narXiv preprint arXiv:1909.05858 (2019).[\ncode\n]\n[15] Marc’Aurelio Ranzato et al.\n“Sequence Level Training with Recurrent Neural Networks.”\nICLR 2016.\n[16] Yonghui Wu et al.\n“Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.”\nCoRR 2016.\n[17] Romain Paulus et al.\n“A Deep Reinforced Model for Abstractive Summarization.”\nICLR 2018.\n[18] Paul Christiano et al.\n“Deep Reinforcement Learning from Human Preferences.”\nNIPS 2017.\n[19] Sanghyun Yi et al.\n“Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators.”\nINLG 2019.\n[20] Florian Böhm et al.\n“Better rewards yield better summaries: Learning to summarise without references.”\nEMNLP 2019. [\ncode\n]\n[21] Daniel M Ziegler et al.\n“Fine-tuning language models from human preferences.”\narXiv preprint arXiv:1909.08593 (2019). [\ncode\n]\n[22] Nisan Stiennon, et al.\n“Learning to summarize from human feedback.”\narXiv preprint arXiv:2009.01325 (2020).\n[23] Sumanth Dathathri et al.\n“Plug and play language models: a simple approach to controlled text generation.”\nICLR 2020. [\ncode\n]\n[24] Jeffrey O Zhang et al.\n“Side-tuning: Network adaptation via additive side networks”\nECCV 2020.\n[25] Ben Kruse et al.\n“GeDi: Generative Discriminator Guided Sequence Generation.”\narXiv preprint arXiv:2009.06367.\n[26] Yoel Zeldes et al.\n“Technical Report: Auxiliary Tuning and its Application to Conditional Text Generatio.”\narXiv preprint arXiv:2006.16823.\n[27] Thomas Scialom, et al.\n“Discriminative Adversarial Search for Abstractive Summarization”\nICML 2020.\n[28] Clara Meister, et al.\n“If beam search is the answer, what was the question?”\nEMNLP 2020.\n[29] Xiang Lisa Li and Percy Liang.\n“Prefix-Tuning: Optimizing Continuous Prompts for Generation.”\narXiv preprint arXiv:2101.00190 (2021).\n[30] Lianhui Qin, et al.\n“Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning.”\narXiv preprint arXiv:2010.05906 (2020).\n[31] Muhammad Khalifa, et al.\n“A Distributional Approach to Controlled Text Generation”\nAccepted by ICLR 2021.\n[32] Aditya Grover, et al.\n“Bias correction of learned generative models using likelihood-free importance weighting.”\nNeuriPS 2019.\n[33] Yuntian Deng et al.\n“Residual Energy-Based Models for Text Generation.”\nICLR 2020.\n[34] Brian Lester et al.\n“The Power of Scale for Parameter-Efficient Prompt Tuning.”\narXiv preprint arXiv:2104.08691 (2021).\n[35] Xiao Liu et al.\n“GPT Understands, Too.”\narXiv preprint arXiv:2103.10385 (2021).\n[36] Welleck & Kulikov et al.\n“Neural Text Generation with Unlikelihood Training”\narXiv:1908.04319 (2019).",
  "metadata": {
    "title": "Controllable Neural Text Generation | Lil'Log",
    "description": "[Updated on 2021-02-01: Updated to version 2.0 with several work added and many typos fixed.]\n\n[Updated on 2021-05-26: Add P-tuning and Prompt Tuning in the “prompt design” section.]\n\n[Updated on 2021-09-19: Add “unlikelihood training”.]",
    "keywords": "nlp, language-model, alignment, steerability, reinforcement-learning, long-read",
    "author": "Lilian Weng",
    "og:title": "Controllable Neural Text Generation",
    "og:description": "[Updated on 2021-02-01: Updated to version 2.0 with several work added and many typos fixed.]\n\n[Updated on 2021-05-26: Add P-tuning and Prompt Tuning in the “prompt design” section.]\n\n[Updated on 2021-09-19: Add “unlikelihood training”.]",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/",
    "twitter:card": "summary",
    "twitter:title": "Controllable Neural Text Generation",
    "twitter:description": "[Updated on 2021-02-01: Updated to version 2.0 with several work added and many typos fixed.]\n\n[Updated on 2021-05-26: Add P-tuning and Prompt Tuning in the “prompt design” section.]\n\n[Updated on 2021-09-19: Add “unlikelihood training”.]",
    "article:section": "posts",
    "article:published_time": "2021-01-02T00:00:00+00:00",
    "article:modified_time": "2021-01-02T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/91c8f7fce27f/code/body-000.txt",
      "chars": 277,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/beam_search_less_surprising.png",
      "bytes": 357907,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-d2b6322a01096e6af110.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/beam_search_less_surprising.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/beam-search-greedy-regularizer.png",
      "bytes": 349638,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-9f26fe897c515951b146.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/beam-search-greedy-regularizer.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/beam-search-size-regularized.png",
      "bytes": 505465,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-9957ed2f40e5f8cee040.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/beam-search-size-regularized.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/SIR-importance-resampling.png",
      "bytes": 116273,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-0663f02bc5222c5ac71d.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/SIR-importance-resampling.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/top-k-joint-sampling.png",
      "bytes": 108086,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-739a21f8c03e3376e149.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/top-k-joint-sampling.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/autoprompt.png",
      "bytes": 197550,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-50d70e010de48d14bfcd.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/autoprompt.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/universal-adv-triggers.png",
      "bytes": 259960,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-81f5d2032bb6b55bebe7.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/universal-adv-triggers.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/autoprompt-examples.png",
      "bytes": 371886,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-957b6cc51fee5bdee28a.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/autoprompt-examples.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prefix-tuning.png",
      "bytes": 322246,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-09af526dca703b251e6d.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prefix-tuning.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prefix-tuning-length.png",
      "bytes": 210135,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-f1c747cea99d0ce61cef.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prefix-tuning-length.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/p-tuning.png",
      "bytes": 165856,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-c84da34a57b51b86561c.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/p-tuning.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prompt-tuning.png",
      "bytes": 79129,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-b60b30dab4c7aabed792.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prompt-tuning.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prompt-tuning-exp1.png",
      "bytes": 386164,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-00fd3a1040622bd96c07.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prompt-tuning-exp1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prompt-tuning-exp2.png",
      "bytes": 142791,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-d89b90d111c8eb41b9e1.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prompt-tuning-exp2.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prompt-small-modifications.png",
      "bytes": 68950,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-b7a5af758c036f2db8d5.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/prompt-small-modifications.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/CTRL-control-code.png",
      "bytes": 388593,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-7eee6bfd8adf3fce09c9.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/CTRL-control-code.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/CTRL-examples.png",
      "bytes": 389939,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-9686fc08f340fe919797.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/CTRL-examples.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/finetune-human-feedback.png",
      "bytes": 214432,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-38aa3d2ff360d669ba9c.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/finetune-human-feedback.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/summarize-human-feedback.png",
      "bytes": 515040,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-e29a62f704968c5029fd.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/summarize-human-feedback.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/PPLM.png",
      "bytes": 255396,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-7c7cff19bd4e438fa515.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/PPLM.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/PPLM-examples.png",
      "bytes": 481960,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-06b73d2d02f24395d5d7.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/PPLM-examples.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/side-tuning.png",
      "bytes": 398304,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-ee944dd3db631ce119b1.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/side-tuning.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/side-auxiliary.png",
      "bytes": 579155,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-79addb31faa52dac417a.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/side-auxiliary.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/GeDi.png",
      "bytes": 273561,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-930398cddc87a89337ab.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/GeDi.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/GDC-KL-adaptive-DPG.png",
      "bytes": 187310,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-17555a40940c559ba50c.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/GDC-KL-adaptive-DPG.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/GDC-debiasing.png",
      "bytes": 242851,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-03c3b856b009d3062779.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/GDC-debiasing.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/GDC-ablation.png",
      "bytes": 941944,
      "reused": false,
      "local_path": "attachments/91c8f7fce27f/images/body-642f3193b6aae322b01f.png",
      "src": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/GDC-ablation.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://arxiv.org/abs/1904.09751",
      "label": "Holtzman et al. 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1805.04833",
      "label": "Fan et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1909.05858",
      "label": "Keskar et al. 2019",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P17-4008/",
      "label": "Ghazvininejad et al., 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1805.06087",
      "label": "Holtzman et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1701.06549",
      "label": "Li et al., 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1809.01215",
      "label": "Baheti et al. (2018)",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Cooperative_principle",
      "label": "Grice’s maxims",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2002.10375",
      "label": "Scialom et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.02650",
      "label": "Meister et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1702.02429",
      "label": "Gu et al (2017)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1605.03835",
      "label": "NPAD",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1906.09531",
      "label": "Grover et al. (2019)",
      "type": "a"
    },
    {
      "href": "https://svivek.com/teaching/lectures/slides/prob-learning/bayes-optimal-classifier.pdf",
      "label": "Bayes optimal",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2004.11714",
      "label": "Deng et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1906.03351",
      "label": "residual space",
      "type": "a"
    },
    {
      "href": "https://www.kdnuggets.com/2019/07/introduction-noise-contrastive-estimation.html",
      "label": "NCE",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2019-01-31-lm/",
      "label": "GPT2",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.15980",
      "label": "Shin et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1911.12543",
      "label": "Jiang et al., 2020)",
      "type": "a"
    },
    {
      "href": "http://ucinlp.github.io/autoprompt",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1908.07125",
      "label": "Wallace et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2101.00190",
      "label": "Li & Liang (2021)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2103.10385",
      "label": "Liu et al. 2021",
      "type": "a"
    },
    {
      "href": "https://github.com/THUDM/P-tuning",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2104.08691",
      "label": "Lester et al. 2021",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P18-1079/",
      "label": "Ribeiro et al (2018)",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/W18-1505/",
      "label": "Peng et al (2018)",
      "type": "a"
    },
    {
      "href": "https://github.com/salesforce/ctrl",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1511.06732",
      "label": "Ranzato et al., 2015",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1609.08144",
      "label": "Wu et al., 2016",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1707.07402",
      "label": "Nguyen et al., 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1705.04304",
      "label": "Paulus et al., 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1804.07036",
      "label": "Wu and Hu, 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1809.10736",
      "label": "Tambwekar et al., 2018",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1706.03741",
      "label": "Christiano et al., 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1904.13015",
      "label": "Yi et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1909.01214",
      "label": "Böhm et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1909.08593",
      "label": "Ziegler et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.01325",
      "label": "Stiennon et al., 2020",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-04-08-policy-gradient/",
      "label": "PPO",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1612.00005",
      "label": "Nguyen et al., 2017",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1912.02164",
      "label": "Dathathri et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1809.00125",
      "label": "post-norm fusion",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.05906",
      "label": "Qin et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1912.13503",
      "label": "Zhang et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1709.07871",
      "label": "FiLM",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.16823",
      "label": "Zeldes et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2009.06367",
      "label": "Kruse et al., 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2012.11635",
      "label": "Khalifa, et al. 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1912.08517",
      "label": "Distributional Policy Gradient",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1908.04319",
      "label": "Welleck & Kulikov et al. 2019",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/blog/how-to-generate",
      "label": "“How to generate text: using different decoding methods for language generation with Transformers”",
      "type": "a"
    },
    {
      "href": "https://www.aclweb.org/anthology/P17-4008",
      "label": "“Hafez: an interactive poetry generation system.”",
      "type": "a"
    },
    {
      "href": "https://github.com/Eric-Wallace/universal-triggers",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://github.com/yg211/summary-reward-no-reference",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/lm-human-preferences",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://github.com/uber-research/PPLM",
      "label": "code",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:25Z",
  "source_url": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.09751",
      "bytes": 4691115,
      "reused": true,
      "local_path": "papers/repair/cfae11e8402ce9b1dc45.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1805.04833",
      "bytes": 1213827,
      "reused": false,
      "local_path": "papers/repair/63d2ed791d86e6282f52.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1909.05858",
      "bytes": 397593,
      "reused": false,
      "local_path": "papers/repair/88b09dc46a4672339e3b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1805.06087",
      "bytes": 1773896,
      "reused": false,
      "local_path": "papers/repair/003a4c27bc171d2cebe1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1701.06549",
      "bytes": 373004,
      "reused": false,
      "local_path": "papers/repair/bb3b4c4fc77f93c11428.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1809.01215",
      "bytes": 659769,
      "reused": false,
      "local_path": "papers/repair/6582219f6b29f0f68c87.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2002.10375",
      "bytes": 2333702,
      "reused": false,
      "local_path": "papers/repair/9043c7f2b0b0c4909d8b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.02650",
      "bytes": 1730086,
      "reused": false,
      "local_path": "papers/repair/cbce152a4f17ba3d6604.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1702.02429",
      "bytes": 844136,
      "reused": false,
      "local_path": "papers/repair/d75933649c2015860122.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1605.03835",
      "bytes": 131168,
      "reused": false,
      "local_path": "papers/repair/cfe9bad7ab372ede814f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1906.09531",
      "bytes": 870301,
      "reused": false,
      "local_path": "papers/repair/bee47c555bbda80764f5.pdf"
    },
    {
      "status": "ok",
      "url": "https://svivek.com/teaching/lectures/slides/prob-learning/bayes-optimal-classifier.pdf",
      "bytes": 345802,
      "reused": false,
      "local_path": "papers/repair/fb2b5b680ab4737aeecd.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2004.11714",
      "bytes": 1065971,
      "reused": false,
      "local_path": "papers/repair/bc96a577677d2a123ab7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1906.03351",
      "bytes": 524736,
      "reused": false,
      "local_path": "papers/repair/dd9a654068c527c8815c.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.15980",
      "bytes": 663180,
      "reused": true,
      "local_path": "papers/repair/5fc241cafafb7eeca9db.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1911.12543",
      "bytes": 476941,
      "reused": false,
      "local_path": "papers/repair/0e13c9a4e6069ed761ca.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1908.07125",
      "bytes": 442623,
      "reused": true,
      "local_path": "papers/repair/a4d5a42a5cf1b3e48ede.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2101.00190",
      "bytes": 1569113,
      "reused": true,
      "local_path": "papers/repair/a704430cffe77134ac6b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2103.10385",
      "bytes": 1583446,
      "reused": true,
      "local_path": "papers/repair/4ed2a5a05e751b4a663d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2104.08691",
      "bytes": 547768,
      "reused": true,
      "local_path": "papers/repair/2ac4e46403c986515128.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1511.06732",
      "bytes": 1713690,
      "reused": false,
      "local_path": "papers/repair/ae6825b749031ce49fc1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1609.08144",
      "bytes": 1688289,
      "reused": false,
      "local_path": "papers/repair/f8e7010861b5c08375c8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1707.07402",
      "bytes": 1352150,
      "reused": false,
      "local_path": "papers/repair/4cbaaa353724bc17b312.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1705.04304",
      "bytes": 460182,
      "reused": false,
      "local_path": "papers/repair/c1949df9b95485a133b7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1804.07036",
      "bytes": 346824,
      "reused": false,
      "local_path": "papers/repair/004326b1927a16142481.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1809.10736",
      "bytes": 334692,
      "reused": false,
      "local_path": "papers/repair/512700a73308ddeb0c56.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1706.03741",
      "bytes": 3218596,
      "reused": true,
      "local_path": "papers/repair/06487c88cb70eb4ab099.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1904.13015",
      "bytes": 594516,
      "reused": false,
      "local_path": "papers/repair/5e6847d24e65dce1e4d7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1909.01214",
      "bytes": 366793,
      "reused": false,
      "local_path": "papers/repair/5a89df59822679391150.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1909.08593",
      "bytes": 967470,
      "reused": true,
      "local_path": "papers/repair/6e863e9d0bbb0e25e44f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.01325",
      "bytes": 1670544,
      "reused": true,
      "local_path": "papers/repair/2b4f670aa5138d617d3b.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1612.00005",
      "bytes": 25726389,
      "reused": false,
      "local_path": "papers/repair/cc4edcb061cdc34f66a2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1912.02164",
      "bytes": 589474,
      "reused": true,
      "local_path": "papers/repair/90a96ef437071e1f745a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1809.00125",
      "bytes": 505180,
      "reused": false,
      "local_path": "papers/repair/fc2a401c997dcb790087.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.05906",
      "bytes": 5416369,
      "reused": false,
      "local_path": "papers/repair/3886cab4b94f1d14ed04.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1912.13503",
      "bytes": 1009911,
      "reused": false,
      "local_path": "papers/repair/2ffdab9ec07ac13e2ed2.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1709.07871",
      "bytes": 6415999,
      "reused": false,
      "local_path": "papers/repair/64041d3291bfa5c0cc26.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.16823",
      "bytes": 402319,
      "reused": false,
      "local_path": "papers/repair/168e1355fe99b2a085bd.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2009.06367",
      "bytes": 1361683,
      "reused": true,
      "local_path": "papers/repair/5368b60783b2cc11e761.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2012.11635",
      "bytes": 4108382,
      "reused": false,
      "local_path": "papers/repair/8acb8ff8ad5cb72236c6.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1912.08517",
      "bytes": 1367530,
      "reused": false,
      "local_path": "papers/repair/68656a5d7fb76a96f891.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1908.04319",
      "bytes": 567247,
      "reused": false,
      "local_path": "papers/repair/9d24f747847ebfdc89e2.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 62068,
  "expected_body_images": 27,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/91c8f7fce27f/pages/source.html",
  "local_html": "attachments/91c8f7fce27f/pages/article.html",
  "body_base_path": "attachments/91c8f7fce27f/pages/",
  "body_markdown_path": "attachments/91c8f7fce27f/pages/article.md"
}