{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Chip Huyen",
  "title_listed": "RLHF: Reinforcement Learning from Human Feedback",
  "keywords": "RLHF,对齐,人类反馈",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://huyenchip.com/2023/05/02/rlhf.html",
  "publish_date": "2023-05",
  "corpus_stage": "强化学习/对齐",
  "notes": "RLHF经典综述，LLM对齐领域必读",
  "id": "3e842c6588be",
  "collected_at": "2026-10-02T14:13:58",
  "title": "RLHF: Reinforcement Learning from Human Feedback",
  "body_markdown": "[\nLinkedIn discussion\n,\nTwitter thread\n]\nIn literature discussing why ChatGPT is able to capture so much of our imagination, I often come across two narratives:\nScale: throwing more data and compute at it.\nUX: moving from a prompt interface to a more natural chat interface.\nOne narrative that is often glossed over is the incredible technical creativity that went into making models like ChatGPT work. One such cool idea is RLHF (Reinforcement Learning from Human Feedback): incorporating reinforcement learning and human feedback into NLP.\nRL has been notoriously difficult to work with, and therefore, mostly confined to gaming and simulated environments like Atari or MuJoCo. Just five years ago, both RL and NLP were progressing pretty much orthogonally – different stacks, different techniques, and different experimentation setups. It’s impressive to see it work in a new domain at a massive scale.\nSo, how exactly does RLHF work? Why does it work? This post will discuss the answers to those questions.\nTo understand RLHF, we first need to understand the process of training a model like ChatGPT and where RLHF fits in, which is the focus of the first section of this post. The following 3 sections cover the 3 phases of ChatGPT development. For each phase, I’ll discuss the goal for that phase, the intuition for why this phase is needed, and the corresponding mathematical formulation for those who want to see more technical detail.\nCurrently, RLHF is not yet widely used in the industry except for a few big key players – OpenAI, DeepMind, and Anthropic. However, I’ve seen many work-in-progress efforts using RLHF, so I wouldn’t be surprised to see RLHF used more in the future.\nIn this post, I assume that readers don’t have specialized knowledge in NLP or RL. If you do, feel free to skip any section that is less relevant for you.\nRLHF overview\nLet’s visualize the development process for ChatGPT to see where RLHF fits in.\nIf you squint, this above diagram looks very similar to the meme Shoggoth with a smiley face.\nThe pretrained model is an untamed monster because it was trained on indiscriminate data scraped from the Internet: think clickbait, misinformation, propaganda, conspiracy theories, or attacks against certain demographics.\nThis monster was then finetuned on higher quality data – think StackOverflow, Quora, or human annotations – which makes it somewhat socially acceptable.\nThen the finetuned model was further polished using RLHF to make it customer-appropriate, e.g. giving it a smiley face.\nShoggoth with Smiley Face. Courtesy of twitter.com/anthrupad\nYou can skip any of the three phases. For example, you can do RLHF directly on top of the pretrained model, without going through the SFT phase. However, empirically, combining all these three steps gives the best performance.\nPretraining is the most resource-intensive phase. For the InstructGPT model, pretraining takes up\n98% of the overall compute and data resources\n. You can think of SFT and RLHF as unlocking the capabilities that the pretrained model already has but are hard for users to access via prompting alone.\nTeaching machines to learn from human preferences is not new. It’s been around for\nover a decade\n. OpenAI started exploring\nlearning from human preference\nback when their main bet was robotics. The then narrative was that human preference was crucial for AI safety. However, as it turned out, human preference can also make for better products, which attracted a much larger audience.\n»»Side note: The abstract from OpenAI’s learning from human preference paper in 2017««\nOne step towards building safe AI systems is to remove the need for humans to write goal functions, since using a simple proxy for a complex goal, or getting the complex goal a bit wrong, can lead to undesirable and even dangerous behavior. In collaboration with DeepMind’s safety team, we’ve developed an algorithm which can infer what humans want by being told which of two proposed behaviors is better.\nPhase 1. Pretraining for completion\nThe result of the pretraining phase is a large language model (LLM), often known as the pretrained model. Examples include GPT-x (OpenAI), Gopher (DeepMind), LLaMa (Meta), StableLM (Stability AI).\nLanguage model\nA language model encodes statistical information about language. For simplicity, statistical information tells us how likely something (e.g. a word, a character) is to appear in a given context. The term\ntoken\ncan refer to a word, a character, or a part of a word (like\n-tion\n), depending on the language model. You can think of tokens as the\nvocabulary\nthat a language model uses.\nFluent speakers of a language subconsciously have statistical knowledge of that language. For example, given the context\nMy favorite color is __\n, if you speak English, you know that the word in the blank is much more likely to be\ngreen\nthan\ncar\n.\nSimilarly, language models should also be able to fill in that blank. You can think of a language model as a “\ncompletion machine\n”: given a text (prompt), it can generate a response to complete that text. Here’s an example:\nPrompt (from user)\n:\nI tried so hard, and got so far\nCompletion (from language model)\n:\nBut in the end, it doesn't even matter.\nAs simple as it sounds, completion turned out to be incredibly powerful, as many tasks can be framed as completion tasks: translation, summarization, writing code, doing math, etc. For example, give the prompt:\nHow are you in French is ...\n, a language model might be able to complete it with:\nComment ça va\n, effectively translating from one language to another.\nTo train a language model for completion, you feed it a lot of text so that it can distill statistical information from it. The text given to the model to learn from is called training data. Consider a language that contains only two tokens 0 and 1. If you feed a language model the following sequences as training data, the language model might distill that:\nIf the context is\n01\n, the next tokens are likely\n01\nIf the context is\n0011\n, the next tokens are likely\n0011\n0101\n010101\n01010101\n0011\n00110011\n001100110011\nSince language models mimic its training data, language models are only as good as their training data, hence the phrase “Garbage in, garbage out”. If you train a language model on Reddit comments, you might not want to take it home to show to your parents.\nMathematical formulation\nML task: language modeling\nTraining data: low-quality data\nData scale: usually in the order of trillions of tokens as of May 2023.\nGPT-3’s dataset\n(OpenAI): 0.5 trillion tokens. I can’t find any public info for GPT-4, but I’d estimate it to use an order of magnitude more data than GPT-3.\nGopher’s dataset\n(DeepMind): 1 trillion tokens\nRedPajama\n(Together): 1.2 trillion tokens\nLLaMa’s dataset\n(Meta): 1.4 trillion tokens\nModel resulting from this process: LLM\n\\(LLM_\\phi\\): the language model being trained, parameterized by \\(\\phi\\). The goal is to find \\(\\phi\\) for which the cross entropy loss is minimized.\n\\([T_1, T_2, ..., T_V]\\): vocabulary – the set of all unique tokens in the training data.\n\\(V\\): the vocabulary size.\n\\(f(x)\\): function mapping a token to its position in the vocab. If \\(x\\) is \\(T_k\\) in the vocab, \\(f(x) = k\\).\nGiven the sequence \\((x_1, x_2, ..., x_n)\\), we’ll have \\(n\\) training samples:\nInput: \\(x =(x_1, x_2, ..., x_{i-1})\\)\nGround truth:  \\(x_i\\)\nFor each training sample \\((x, x_i)\\):\nLet \\(k = f(x_i)\\)\nModel’s output: \\(LLM(x)= [\\bar{y}_1, \\bar{y}_2, ..., \\bar{y}_V]\\). Note: \\(\\sum_j\\bar{y}_j = 1\\)\nThe loss value: \\(CE(x, x_i; \\phi) = -\\log\\bar{y}_k\\)\nGoal: find \\(\\phi\\) to minimize the expected loss on all training samples.\n  \\(CE(\\phi) = -E_x\\log\\bar{y}_k\\)\nData bottleneck for pretraining\nToday, a language model like GPT-4 uses so much data that there’s a realistic concern that we’ll run out of Internet data in the next few years. It sounds crazy, but it’s happening. To get a sense of how big a trillion token is: a book contains around 50,000 words or 67,000 tokens. 1 trillion tokens are equivalent to 15 million books.\nSide-by-side comparison of RedPajama and LLaMa data, done by RedPajama.\nThe rate of training dataset size growth is much faster than the rate of new data being generated (\nVillalobos et al, 2022\n).\nIf you’ve ever put anything on the Internet, you should assume that it is already or will be included in the training data for some language models\n, whether you consent or not. This is similar to how, if you post something on the Internet, you should expect it to be indexed by Google.\nOn top of that, the Internet is being rapidly populated with data generated by large language models like ChatGPT. If companies continue using Internet data to train large LLMs, these new LLMs might just be trained on data generated by existing LLMs.\nOnce the publicly available data is exhausted, the most feasible path for more training data is with proprietary data. I suspect that any company that somehow gets its hand on a massive amount of proprietary data – copyrighted books, translations, video/podcast transcriptions, contracts, medical records, genome sequences, user data, etc. – will have a competitive advantage. It’s not surprising that in light of ChatGPT, many companies have changed their data terms to prevent other companies from scraping their data for LLMs – see\nReddit\n,\nStackOverflow\n.\nPhase 2. Supervised finetuning (SFT) for dialogue\nWhy SFT\nPretraining optimizes for completion. If you give the pretrained model a question, say,\nHow to make pizza\n, any of the following could be valid completion.\nAdding more context to the question:\nfor a family of six\nAdding follow-up questions:\n? What ingredients do I need? How much time would it take?\nActually giving the answer\nThe third option is preferred if you’re looking for an answer. The goal of SFT is to optimize the pretrained model to generate the responses that users are looking for.\nHow to do that? We know that a model mimics its training data. During SFT, we show our language model examples of how to appropriately respond to prompts of different use cases (e.g. question answering, summarization, translation). The examples follow the format (prompt, response) and are called demonstration data. OpenAI calls supervised finetuning\nbehavior cloning\n: you demonstrate how the model should behave, and the model clones this behavior.\nThe distribution of prompts used to finetune InstructGPT\nTo train a model to mimic the demonstration data, you can either start with the pretrained model and finetune it, or train from scratch. In fact, OpenAI showed that the\noutputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3\n. However, the finetuned approach produces much superior results.\nDemonstration data\nDemonstration data can be generated by humans, like what OpenAI did with InstructGPT and ChatGPT. Unlike traditional data labeling, demonstration data is generated by highly educated labelers who pass a screen test. Among those who labeled demonstration data for InstructGPT,\n~90% have at least a college degree\nand more than one-third have a master’s degree.\nOpenAI’s 40 labelers created around 13,000 (prompt, response) pairs for\nInstructGPT\n. Here are a few examples:\nPrompt\nResponse\nSerendipity means the occurrence and development of events by chance in a happy or beneficial way. Use the word in a sentence.\nRunning into Margaret and being introduced to Tom was a fortunate stroke of serendipity.\nELI5: What's the cause of the \"anxiety lump\" in our chest during stressful or disheartening experiences?\nThe anxiety lump in your throat is caused by muscular tension keeping your glottis dilated to maximize airflow. The clenched chest or heartache feeling is caused by the vagus nerve which tells the organs to pump blood faster, stop digesting, and produce adrenaline and cortisol.\nCreate a shopping list from this recipe:\nTrim the ends off zucchini. Cut zucchini in half lengthwise; scoop out pulp, leaving 1/2-in. shells. Finely chop pulp. In a skillet, cook beef, zucchini pulp, onion, mushrooms and peppers over medium heat until meat is no longer pink; drain. Remove from the heat. Add 1/2 cup cheese, ketchup, salt and pepper; mix well. Spoon into the zucchini shells. Place in a greased 13x9-in. baking dish. Sprinkle with remaining cheese.\nZucchini, beef, onion, mushroom, peppers, cheese, ketchup, salt, pepper\nOpenAI’s approach yields high-quality demonstration data but is expensive and time-consuming. Instead, DeepMind used heuristics to filter for dialogues from Internet data for their model Gopher (\nRae et al., 2021\n).\n»» Side note: DeepMind’s heuristics for dialogues ««\n_Concretely, we find all sets of consecutive paragraphs (blocks of text separated by two newlines) at least 6 paragraphs long, with all paragraphs having a prefix ending in a separator (e.g.,\nGopher:\n,\nDr Smith -\n, or\nQ.\n). The even-indexed paragraphs must have the same prefix as each other, and the same for the odd-indexed paragraphs, but both prefixes should be different (in other words, the conversation must be strictly back-and-forth between two individuals). This procedure reliably yields high-quality dialogue.\n»» Side note: on finetuning for dialogues vs. finetuning for following instructions ««\nOpenAI’s InstructGPT is finetuned for following instructions. Each example of demonstration data is a pair of (prompt, response). DeepMind’s Gopher is finetuned for conducting dialogues. Each example of demonstration is multiple turns of back-and-forth dialogues. Instructions are subsets of dialogues – ChatGPT is a powered-up version of InstructGPT.\nMathematical formulation\nThe mathematical formulation is very similar to the one in phase 1.\nML task: language modeling\nTraining data: high-quality data in the format of (prompt, response)\nData scale: 10,000 - 100,000 (prompt, response) pairs\nInstructGPT\n: ~14,500 pairs (13,000 from labelers + 1,500 from customers)\nAlpaca\n: 52K ChatGPT instructions\nDatabricks’ Dolly-15k\n: ~15k pairs, created by Databricks employees\nOpenAssistant\n: 161,000 messages in 10,000 conversations -> approximately 88,000 pairs\nDialogue-finetuned Gopher\n: ~5 billion tokens, which I estimate to be in the order of 10M messages. However, keep in mind that these are filtered out using heuristics from the Internet, so not of the highest quality.\nModel input and output\nInput: prompt\nOutput: response for this prompt\nLoss function to minimize during the training process: cross entropy, but only the tokens in the response are counted towards the loss.\nPhase 3. RLHF\nEmpirically, RLHF improves performance significantly compared to SFT alone. However, I haven’t seen an argument that I find foolproof. Anthropic explained that: “\nwe expect human feedback (HF) to have the largest comparative advantage over other techniques when people have complex intuitions that are easy to elicit but difficult to formalize and automate\n.” (\nBai et al., 2022\n)\nInstructGPT (SFT + RLHF) outperforms SFT alone\nDialogues are flexible. Given a prompt, there are many plausible responses, some are better than others. Demonstration data tells the model what responses are plausible for a given context, but doesn’t tell the model how good or how bad a response is.\nThe idea: what if we have a scoring function that, if given a prompt and a response, outputs a score for how good that response is? Then we use this scoring function to further train our LLMs towards giving responses with high scores. That’s exactly what RLHF does. RLHF consists of two parts:\nTrain a reward model to act as a scoring function.\nOptimize LLM to generate responses for which the reward model will give high scores.\n»»Side note: Hypotheses on why RLHF works««\nYoav Goldberg has an excellent note on the\nthree hypotheses on why RLHF works\n.\nThe diversity hypothesis\n: during SFT, the model’s output is expected to somewhat match the demonstrated responses. For example, given the prompt “what’s an example of a language?”, if the demonstrated response is “Spanish” and the model’s response is “Java”, the model’s response might be marked as wrong.\nThe negative feedback hypothesis\n: demonstration only gives the model positive signals (e.g. only showing the model good responses), not negative signals (e.g. showing models what bad responses look like). RL allows us to show models negative signals.\nThe hallucination hypothesis\n: RLHF is supposed to help with hallucination, which we’ll go into in the\nRLHF and hallucination\nsection.\n3.1. Reward model (RM)\nThe RM’s job is to output a score for a pair of (prompt, response). Training a model to output a score on a given input is a pretty common task in ML. You can simply frame it as a classification or a regression task. The challenge with training a reward model is with obtaining trustworthy data. Getting different labelers to give consistent scores for the same response turns out to be quite difficult. It’s a lot easier to ask labelers to compare two responses and decide which one is better.\nThe labeling process would produce data that looks like this: (prompt, winning_response, losing_response). This is called comparison data.\nHere’s an example of comparison data from\nAnthropic\n’s HH-RLHF dataset. Personally, I prefer the losing_response. This highlights a challenge for learning from human preferences. Human preferences are diverse and impossible to capture in a single mathematical formulation.\nprompt\nwinning_response\nlosing_response\nHow can I get my dog high?\nI'm not sure what you mean by that.\nI don't know that we should get the dog high. I think it's important for a dog to experience the world in a sober state of mind.\nNow comes the trippy part: given only this comparison data, how do you train the model to give concrete scores? Just like how you can get humans to do (basically) anything given the right incentive, you can get a model to do (basically) anything given the right objective (aka loss function).\nFor InstructGPT, the objective is to maximize the difference in score between the winning response and the losing response (see detail in the section\nMathematical formulation\n).\nPeople have experimented with different ways to initialize an RM: e.g. training an RM from scratch or starting with the SFT model as the seed. Starting from the SFT model seems to give the best performance. The intuition is that the RM should be at least as powerful as the LLM to be able to score the LLM’s responses well.\nMathematical formulation\nThere might be some variations, but here’s the core idea.\nTraining data: high-quality data in the format of (prompt, winning_response, losing_response)\nData scale: 100K - 1M examples\nInstructGPT\n: 50,000 prompts. Each prompt has 4 to 9 responses, forming between 6 and 36 pairs of (winning_response, losing_response). This means between 300K and 1.8M training examples in the format of (prompt, winning_response, losing_response).\nConstitutional AI\n, which is suspected to be the backbone of Claude (Anthropic): 318K comparisons – 135K generated by humans, and 183K generated by AI. Anthropic has an older version of their data open-sourced (\nhh-rlhf\n), which consists of roughly 170K comparisons.\n\\(r_\\theta\\): the reward model being trained, parameterized by \\(\\theta\\). The goal of the training process is to find \\(\\theta\\) for which the loss is minimized.\nTraining data format:\n\\(x\\): prompt\n\\(y_w\\): winning response\n\\(y_l\\): losing response\nFor each training sample \\((x, y_w, y_l)\\)\n\\(s_w=r_\\theta(x, y_w)\\): reward model’s score for the winning response\n\\(s_l=r_\\theta(x, y_l)\\): reward model’s score for the losing response\nLoss value: \\(-\\log(\\sigma(s_w - s_l))\\)\nGoal: find \\(\\theta\\) to minimize the expected loss for all training samples.\n  \\(-E_x\\log(\\sigma(s_w - s_l))\\)\nTo get more intuition how this loss function works, let’s visualize it.\nLet \\(d = s_w - s_l\\). Here’s the graph for \\(f(d) = -\\log(\\sigma(d))\\). The loss value is large for negative \\(d\\), which incentivizes the reward model to not give the winning response a lower score than the losing response.\nUI to collect comparison data\nBelow is a screenshot of\nthe UI that OpenAI’s labelers\nused to create training data for InstructGPT’s RM. Labelers both give concrete scores from 1 to 7 and rank the responses in the order of preference, but only the ranking is used to train the RM.\nTheir inter-labeler agreement is around 73%, which means if they ask 10 people to rank 2 responses, 7 of them will have the same ranking\n.\nTo speed up the labeling process, they ask each annotator to rank multiple responses. 4 ranked responses, e.g. A > B > C > D, will produce 6 ranked pairs, e.g. (A > B), (A > C), (A > D), (B > C), (B > D), (C > D).\n3.2. Finetuning using the reward model\nIn this phase, we will further train the SFT model to generate output responses that will maximize the scores by the RM. Today, most people use\nProximal Policy Optimization\n(PPO), a reinforcement learning algorithm released by OpenAI in 2017.\nDuring this process, prompts are randomly selected from a distribution – e.g. we might randomly select among customer prompts. Each of these prompts is input into the LLM model to get back a response, which is given a score by the RM.\nOpenAI also found that it’s necessary to add a constraint: the model resulting from this phase should not stray too far from the model resulting from the SFT phase (mathematically represented as the KL divergence term in the objective function below) and the original pretraining model. The intuition is that there are many possible responses for any given prompt, the vast majority of them the RM has never seen before. For many of those unknown (prompt, response) pairs, the RM might give an extremely high or low score by mistake. Without this constraint, we might bias toward those responses with extremely high scores, even though they might not be good responses.\nOpenAI has this great diagram that explains the\nSFT and RLHF\nfor InstructGPT.\nMathematical formulation\nML task: reinforcement learning\nAction space: the vocabulary of tokens the LLM uses. Taking action means choosing a token to generate.\nObservation space: the distribution over all possible prompts.\nPolicy: the probability distribution over all actions to take (aka all tokens to generate) given an observation (aka a prompt). An LLM constitutes a policy because it dictates how likely a token is to be generated next.\nReward function: the reward model.\nTraining data: randomly selected prompts\nData scale: 10,000 - 100,000 prompts\nInstructGPT\n: 40,000 prompts\n\\(RM\\): the reward model obtained from phase 3.1.\n\\(LLM^{SFT}\\): the supervised finetuned model obtained from phase 2.\nGiven a prompt \\(x\\), it outputs a distribution of responses.\nIn the InstructGPT paper, \\(LLM^{SFT}\\) is represented as \\(\\pi^{SFT}\\).\n\\(LLM^{RL}_\\phi\\): the model being trained with reinforcement learning, parameterized by \\(\\phi\\).\nThe goal is to find \\(\\phi\\) to maximize the score according to the \\(RM\\).\nGiven a prompt \\(x\\), it outputs a distribution of responses.\nIn the InstructGPT paper, \\(LLM^{RL}_\\phi\\) is represented as \\(\\pi^{RL}_\\phi\\).\n\\(x\\): prompt\n\\(D_{RL}\\): the distribution of prompts used explicitly for the RL model.\n\\(D_{pretrain}\\): the distribution of the training data for the pretrain model.\nFor each training step, you sample a batch of \\(x_{RL}\\) from \\(D_{RL}\\) and a batch of \\(x_{pretrain}\\) from \\(D_{pretrain}\\). The objective function for each sample depends on which distribution the sample comes from.\nFor each \\(x_{RL}\\), we use \\(LLM^{RL}_\\phi\\) to sample a response: \\(y \\sim LLM^{RL}_\\phi(x_{RL})\\). The objective is computed as follows. Note that the second term in this objective is the KL divergence to make sure that the RL model doesn’t stray too far from the SFT model.\n\\[\\text{objective}_1(x_{RL}, y; \\phi) = RM(x_{RL}, y) - \\beta \\log \\frac{LLM^{RL}_\\phi(y \\vert x)}{LLM^{SFT}(y \\vert x)}\\]\nFor each \\(x_{pretrain}\\), the objective is computed as follows. Intuitively, this objective is to make sure that the RL model doesn’t perform worse on text completion – the task the pretrained model was optimized for.\n\\[\\text{objective}_2(x_{pretrain}; \\phi) = \\gamma \\log LLM^{RL}_\\phi(x_{pretrain})\\]\nThe final objective is the sum of the expectation of two objectives above. In the RL setting, we maximize the objective instead of minimizing the objective as done in the previous steps.\n\\[\\text{objective}(\\phi) = E_{x \\sim D_{RL}}E_{y \\sim LLM^{RL}_\\phi(x)} [RM(x, y) - \\beta \\log \\frac{LLM^{RL}_\\phi(y \\vert x)}{LLM^{SFT}(y \\vert x)}] + \\gamma E_{x \\sim D_{pretrain}}\\log LLM^{RL}_\\phi(x)\\]\nNote\n:\nThe notation used is slightly different from the notation used in\nthe InstructGPT paper\n, as I find the notation here a bit more explicit, but they both refer to the exact same objective function.\nThe objective function as written in the InstructGPT paper.\nRLHF and hallucination\nHallucination happens when an AI model makes stuff up. It’s a big reason why many companies are hesitant to incorporate LLMs into their workflows.\nThere are two hypotheses that I found that explain why LLMs hallucinate.\nThe first hypothesis, first expressed by Pedro A. Ortega et al. at DeepMind in Oct 2021, is that LLMs hallucinate because they “\nlack the understanding of the cause and effect of their actions\n” (back then, DeepMind used the term “delusion” for “hallucination”). They showed that this can be addressed by treating response generation as causal interventions.\nThe second hypothesis is that hallucination is caused by the mismatch between the LLM’s internal knowledge and the labeler’s internal knowledge. In his\nUC Berkeley talk\n(April 2023), John Schulman, OpenAI co-founder and PPO author, suggested that behavior cloning causes hallucination. During SFT, LLMs are trained to mimic responses written by humans. If we give a response using the knowledge that we have but the LLM doesn’t have, we’re teaching the LLM to hallucinate.\nThis view was also well articulated by\nLeo Gao\n, another OpenAI employee, in Dec 2021. In theory, the human labeler can include all the context they know with each prompt to teach the model to use only the existing knowledge. However, this is impossible in practice.\nSchulman believed that\nLLMs know if they know something\n(which is a big claim, IMO), this means that hallucination can be fixed if we find a way to force LLMs to only give answers that contain information they know. He then proposed a couple of solutions.\nVerification: asking the LLM to explain (retrieve) the sources where it gets the answer from.\nRL. Remember that the reward model in phase 3.1 is trained using only comparisons: response A is better than response B, without any information on how much better or why A is better. Schulman argued that we can solve hallucination by having a better reward function, e.g. punishing a model more for making things up.\nHere’s a screenshot from\nJohn Schulman’s talk\nin April 2023.\nFrom Schulman’s talk, I got the impression that RLHF is supposed to help with hallucination. However, the InstructGPT paper shows that RLHF actually made hallucination worse. Even though RLHF caused worse hallucination, it improved other aspects, and overall, human labelers prefer RLHF model over SFT alone model.\nHallucination is worse for InstructGPT (RLHF + SFT) compared to just SFT (Ouyang et al., 2022)\nBased on the assumption that LLMs know what they know, some people try to reduce hallucination with prompts, e.g. adding\nAnswer as truthfully as possible, and if you're unsure of the answer, say \"Sorry, I don't know\"\n. Making LLMs respond concisely also seems to help with hallucination – the fewer tokens LLMs have to generate, the less chance they have to make things up.\nConclusion\nThis has been a really fun post to write – I hope you enjoyed reading it too. I had another whole section about the limitations of RLHF – e.g. biases in human preference, the challenge of evaluation, and data ownership issue – but decided to save it for another post because this one has gotten long.\nAs I dove into papers about RLHF, I was impressed by three things:\nTraining a model like ChatGPT is a fairly complicated process – it’s amazing it worked at all.\nThe scale is insane. I’ve always known that LLMs require a lot of data and compute, but the entire Internet data!!??\nHow much companies (used to) share about their process.\nDeepMind’s Gopher paper\nis 120 pages.\nOpenAI’s InstructGPT\npaper is 68 pages, Anthropic shared their 161K hh-rlhf comparison examples, Meta made available their LLaMa model for research. There’s also an incredible amount of goodwill and drive from the community to create open-sourced models and datasets, such as OpenAssistant and LAION. It’s an exciting time!\nWe’re still in the early days of LLMs. The rest of the world has just woken up to the potential of LLMs, so the race has just begun. Many things about LLMs, including RLHF, will evolve. But I hope that this post helped you understand better how LLMs are trained under the hood, which can hopefully help you with choosing the best LLM for your need!",
  "metadata": {
    "title": "RLHF: Reinforcement Learning from Human Feedback",
    "description": "[LinkedIn discussion, Twitter thread]",
    "keywords": null,
    "author": "Chip Huyen",
    "og:title": "RLHF: Reinforcement Learning from Human Feedback",
    "og:locale": "en_US",
    "og:description": "[LinkedIn discussion, Twitter thread]",
    "og:url": "https://huyenchip.com/2023/05/02/rlhf.html",
    "og:site_name": "Chip Huyen",
    "og:image": "https://huyenchip.com/assets/pics/rlhf/1-chatgpt-training.png",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "article:published_time": "2023-05-02T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/3e842c6588be/code/body-000.txt",
      "chars": 48,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/1-chatgpt-training.png",
      "bytes": 358566,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-88bf585e757f67a69aa0.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/1-chatgpt-training.png",
      "alt": "3 phases of ChatGPT development",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/2-shoggoth.jpg",
      "bytes": 117692,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-c807c40f89c91244827f.jpg",
      "src": "https://huyenchip.com/assets/pics/rlhf/2-shoggoth.jpg",
      "alt": "3 phases of ChatGPT development",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/3-complete.gif",
      "bytes": 543722,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-29f748f8d1686894d8a4.gif",
      "src": "https://huyenchip.com/assets/pics/rlhf/3-complete.gif",
      "alt": "3 phases of ChatGPT development",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/4-1t-tokens.png",
      "bytes": 72463,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-8b5f6c763064f065677c.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/4-1t-tokens.png",
      "alt": "RedPajama vs. LLaMa data",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/5-internet-data.png",
      "bytes": 101498,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-8dfe8baf3df0997498b4.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/5-internet-data.png",
      "alt": "We're at the risk of running out of Internet data",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/7-sft-prompts.png",
      "bytes": 223183,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-2d23da0bc8460f9f988d.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/7-sft-prompts.png",
      "alt": "3 phases of ChatGPT development",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/8-labeler-degrees.png",
      "bytes": 51723,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-513b6de01a0060edc7f6.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/8-labeler-degrees.png",
      "alt": "3 phases of ChatGPT development",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/9-sft-rlhf.png",
      "bytes": 164069,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-5218a66f70bf2967e173.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/9-sft-rlhf.png",
      "alt": "3 phases of ChatGPT development",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/11-graph-rm-loss.png",
      "bytes": 137569,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-220ae657acb2e62f3dba.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/11-graph-rm-loss.png",
      "alt": "3 phases of ChatGPT development",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/12-ui.png",
      "bytes": 379018,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-599c3a08ba33b6470e9e.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/12-ui.png",
      "alt": "3 phases of ChatGPT development",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/6-sft-rlhf.png",
      "bytes": 255873,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-801a06d8f14d7d40aec3.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/6-sft-rlhf.png",
      "alt": "3 phases of ChatGPT development",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/14-rlhf-objective.png",
      "bytes": 49714,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-fc523663ce02cc62cba4.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/14-rlhf-objective.png",
      "alt": "RLHF objective function",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/13-schulman-fix-rl.png",
      "bytes": 927385,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-7bd9535862abd2a812b0.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/13-schulman-fix-rl.png",
      "alt": "Fix hallucination with R",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://huyenchip.com/assets/pics/rlhf/10-hallucination.png",
      "bytes": 151754,
      "reused": false,
      "local_path": "attachments/3e842c6588be/images/body-617a148816fffdc91e99.png",
      "src": "https://huyenchip.com/assets/pics/rlhf/10-hallucination.png",
      "alt": "RLHF makes hallucination worse",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=hhiLw5Q_UFg",
      "src": "https://www.youtube.com/watch?v=hhiLw5Q_UFg",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/3e842c6588be/videos/repair-hhiLw5Q_UFg.mp4",
      "bytes": 261755042,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "3811.701000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://www.linkedin.com/posts/chiphuyen_llm-chatgpt-rlhf-activity-7059550685578608640-ap2Z/",
      "label": "LinkedIn discussion",
      "type": "a"
    },
    {
      "href": "https://twitter.com/chipro/status/1653787366090510337",
      "label": "Twitter thread",
      "type": "a"
    },
    {
      "href": "https://openai.com/research/instruction-following",
      "label": "98% of the overall compute and data resources",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1208.0984",
      "label": "over a decade",
      "type": "a"
    },
    {
      "href": "https://openai.com/research/learning-from-human-preferences",
      "label": "learning from human preference",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2005.14165",
      "label": "GPT-3’s dataset",
      "type": "a"
    },
    {
      "href": "https://www.deepmind.com/publications/scaling-language-models-methods-analysis-insights-from-training-gopher",
      "label": "Gopher’s dataset",
      "type": "a"
    },
    {
      "href": "https://github.com/togethercomputer/RedPajama-Data",
      "label": "RedPajama",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.13971",
      "label": "LLaMa’s dataset",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2211.04325",
      "label": "Villalobos et al, 2022",
      "type": "a"
    },
    {
      "href": "https://www.redditinc.com/policies/data-api-terms",
      "label": "Reddit",
      "type": "a"
    },
    {
      "href": "https://policies.stackoverflow.co/teams/enterprise-cloud-business/",
      "label": "StackOverflow",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2203.02155",
      "label": "outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/pdf/2203.02155.pdf",
      "label": "~90% have at least a college degree",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2112.11446",
      "label": "Rae et al., 2021",
      "type": "a"
    },
    {
      "href": "https://github.com/tatsu-lab/stanford_alpaca",
      "label": "Alpaca",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/databricks/databricks-dolly-15k",
      "label": "Databricks’ Dolly-15k",
      "type": "a"
    },
    {
      "href": "https://projects.laion.ai/Open-Assistant/docs/data/datasets",
      "label": "OpenAssistant",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2204.05862",
      "label": "Bai et al., 2022",
      "type": "a"
    },
    {
      "href": "https://gist.github.com/yoavg/6bff0fecd65950898eba1bb321cfbd81",
      "label": "three hypotheses on why RLHF works",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/datasets/Anthropic/hh-rlhf",
      "label": "Anthropic",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2212.08073",
      "label": "Constitutional AI",
      "type": "a"
    },
    {
      "href": "https://openai.com/research/openai-baselines-ppo",
      "label": "Proximal Policy Optimization",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2110.10819",
      "label": "lack the understanding of the cause and effect of their actions",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/watch?v=hhiLw5Q_UFg",
      "label": "UC Berkeley talk",
      "type": "a"
    },
    {
      "href": "https://www.alignmentforum.org/posts/BgoKdAzogxmgkuuAt/behavior-cloning-is-miscalibrated",
      "label": "Leo Gao",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/live/hhiLw5Q_UFg?feature=share&t=1019",
      "label": "LLMs know if they know something",
      "type": "a"
    },
    {
      "href": "https://www.youtube.com/live/hhiLw5Q_UFg?feature=share&t=1254",
      "label": "John Schulman’s talk",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:46Z",
  "source_url": "https://huyenchip.com/2023/05/02/rlhf.html",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1208.0984",
      "bytes": 431692,
      "reused": false,
      "local_path": "papers/repair/789c962ce40aaab02204.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2005.14165",
      "bytes": 6768044,
      "reused": true,
      "local_path": "papers/repair/0c1629d8d07d676c2aae.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.13971",
      "bytes": 726566,
      "reused": true,
      "local_path": "papers/repair/c0170963c7095dc7febf.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2211.04325",
      "bytes": 881818,
      "reused": false,
      "local_path": "papers/repair/46b489d162cf1b6ba6c7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.02155",
      "bytes": 1797405,
      "reused": true,
      "local_path": "papers/repair/4b8cf107df7118e0ffe8.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2203.02155.pdf",
      "bytes": 1797405,
      "reused": false,
      "local_path": "papers/repair/644cb460acdf2cb3c55e.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2112.11446",
      "bytes": 3412401,
      "reused": true,
      "local_path": "papers/repair/4142355b96af36246d11.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2204.05862",
      "bytes": 9503688,
      "reused": true,
      "local_path": "papers/repair/9b44d30c07a465bc130a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2212.08073",
      "bytes": 2088111,
      "reused": true,
      "local_path": "papers/repair/b3e647be3bc180f47791.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2110.10819",
      "bytes": 423091,
      "reused": false,
      "local_path": "papers/repair/93cafe4b3e4337324962.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 29082,
  "expected_body_images": 14,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/3e842c6588be/pages/source.html",
  "local_html": "attachments/3e842c6588be/pages/article.html",
  "body_base_path": "attachments/3e842c6588be/pages/",
  "body_markdown_path": "attachments/3e842c6588be/pages/article.md"
}