{ "source_file": "0930", "source_platform": "个人博客", "author": "Lilian Weng / Lil'Log", "title_listed": "Reward Hacking in Reinforcement Learning", "keywords": "奖励黑客,RLHF,对齐,奖励模型", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/", "publish_date": "2024-11", "corpus_stage": "强化学习/对齐", "notes": "深入剖析RL中的奖励黑客问题及其在LLM对齐中的影响", "id": "af1a476c6de0", "collected_at": "2026-10-02T14:12:38", "title": "Reward Hacking in Reinforcement Learning | Lil'Log", "body_markdown": "Reward hacking occurs when a\nreinforcement learning (RL)\nagent\nexploits\nflaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.\nWith the rise of\nlanguage models\ngeneralizing to a broad spectrum of tasks and RLHF becomes a de facto method for alignment training, reward hacking in RL training of language models has become a critical practical challenge. Instances where the model learns to modify unit tests to pass coding tasks, or where responses contain biases that mimic a user’s preference, are pretty concerning and are likely one of the major blockers for real-world deployment of more autonomous use cases of AI models.\nMost of the past work on this topic has been quite theoretical and focused on defining or demonstrating the existence of reward hacking. However, research into practical mitigations, especially in the context of RLHF and LLMs, remains limited. I especially want to call out for more research efforts directed toward understanding and developing mitigation for reward hacking in the future. Hope I will be able to cover the mitigation part in a dedicated post soon.\nBackground\n#\nReward Function in RL\n#\nReward function defines the task, and reward shaping significantly impacts learning efficiency and accuracy in\nreinforcement learning\n. Designing a reward function for an RL task often feels like a ‘dark art’. Many factors contribute to this complexity: How you decompose a big goal into small goals? Is the reward sparse or dense? How you measure the success? Various choices may lead to good or problematic learning dynamics, including unlearnable tasks or hackable reward functions. There is a long history of research on how to do reward shaping in RL.\nFor example, in an\n1999 paper by Ng et al.\n, the authors studied how to modify the reward function in\nMarkov Decision Processes (MDPs)\nsuch that the optimal policy remains unchanged. They found that linear transformation works. Given a MDP $M = (S, A, T, \\gamma, R)$, we want to create a transformed MDP $M’ = (S, A, T, \\gamma, R’)$ where $R’ = R + F$ and $F: S \\times A \\times S \\mapsto \\mathbb{R}$, such that we can guide the learning algorithm to be more efficient. Given a real-valued function $\\Phi: S \\mapsto \\mathbb{R}$, $F$ is a potential-based shaping function if for all $s \\in S - {s_0}, a \\in A, s’ \\in S$:\n$$\nF(s, a, s') = \\gamma \\Phi(s') - \\Phi(s)\n$$\nThis would guarantee that the sum of discounted $F$, $F(s_1, a_1, s_2) + \\gamma F(s_2, a_2, s_3) + \\dots$, ends up being 0. If $F$ is such a potential-based shaping function, it is both\nsufficient\nand\nnecessary\nto ensure $M$ and $M’$ share the same optimal policies.\nWhen $F(s, a, s’) = \\gamma \\Phi(s’) - \\Phi(s)$, and if we further assume that $\\Phi(s_0) = 0$, where $s_0$ is absorbing state, and $\\gamma=1$, and then for all $s \\in S, a \\in A$:\n$$\n\\begin{aligned}\nQ^*_{M'} (s,a) &= Q^*_M(s, a) - \\Phi(s) \\\\\nV^*_{M'} (s,a) &= V^*_M(s, a) - \\Phi(s)\n\\end{aligned}\n$$\nThis form of reward shaping allows us to incorporate heuristics into the reward function to speed up learning without impacting the optimal policy.\nSpurious Correlation\n#\nSpurious correlation or shortcut learning (\nGeirhos et al. 2020\n) in classification task is a concept closely related to reward hacking. Spurious or shortcut features can cause a classifier to fail at learning and generalizing as intended. For example, a binary classifier for distinguishing wolves from huskies may overfit to the presence of a snowy background if all the wolf training images include snow (\nRibeiro et al. 2024\n).\nThe model performs poorly on out-of-distribution (OOD) test sets if it overfits to shortcut features. (Image source:\nGeirhos et al. 2020\n)\nThe\nERM principle\nstates that, since the full data distribution is unknown, minimizing the loss on training data is a reasonable proxy of risk and thus we favor models with the lowest training loss.\nNagarajan et al. (2021)\nstudied the ERM principle and pointed out that ERM needs to rely on all types of informative features, including unreliable spurious features, while attempting to fit the data without constraints. Their experiments showed that ERM would depend on spurious features no matter how easy the task is.\nLet’s Define Reward Hacking\n#\nReward shaping in RL is challenging. Reward hacking occurs when an RL agent exploits flaws or ambiguities in the reward function to obtain high rewards without genuinely learning the intended behaviors or completing the task as designed. In recent years, several related concepts have been proposed, all referring to some form of reward hacking:\nReward hacking (\nAmodei et al., 2016\n)\nReward corruption (\nEveritt et al., 2017\n)\nReward tampering (\nEveritt et al. 2019\n)\nSpecification gaming (\nKrakovna et al., 2020\n)\nObjective robustness (\nKoch et al. 2021\n)\nGoal misgeneralization (\nLangosco et al. 2022\n)\nReward misspecifications (\nPan et al. 2022\n)\nThe concept originated with Amodei et al. (2016), who proposed a set of open research questions on AI safety in their seminal paper\n“Concrete Problems in AI Safety”\n. They listed\nreward hacking\nas one of the key AI safety problems. Reward hacking refers to the possibility of the agent gaming the reward function to achieve high reward through undesired behavior.\nSpecification gaming\n(\nKrakovna et al. 2020\n) is a similar concept, defined as a behavior that satisfies the literal specification of an objective but not achieving the desired results. Here the literal description of the task goal and the intended goal may have a gap.\nReward shaping is a technique used to enrich the reward function, making it easier for the agent to learn—for example, by providing denser rewards. However, a poorly design reward shaping mechanism can alter the trajectory of the optimal policy. Designing effective reward shaping mechanisms is inherently difficult. Rather than blaming a poorly designed reward function, it is more accurate to acknowledge that designing a good reward function is intrinsically challenging due to the complexity of the task itself, partial observable state, multiple dimensions in consideration, and other factors.\nWhen testing an RL agent in out-of-distribution (OOD) environments, robustness failure may occur due to:\nThe model fails to generalize effectively, even with the right objective. This happens when the algorithm lacks sufficient intelligence or capability.\nThe model generalizes capably but pursues an objective different from the one it was trained on. This happens when the proxy reward differs from the true reward function, $R’ \\neq R$. This is known as\nobjective robustness\n(\nKoch et al. 2021\n) or\ngoal misgeneralization\n(\nLangosco et al. 2022\n)\nExperiments in two RL environments,\nCoinRun\nand\nMaze\n, demonstrated the importance of randomization during training. If during training, the coin or the cheese is placed at a fixed position (i.e. right end of the level or upper right corner of the maze) but testing in the env where the coin or cheese is placed at random, the agent would just run to the fixed position without obtaining the coin or cheese at test time. A conflict arises when a visual feature (e.g., cheese or coin) and a positional feature (e.g., upper-right or right end) are inconsistent during test time, leading the trained model to prefer the positional feature. I would like to point out that, in these two examples, the\nreward-result gaps\nare clear but such type of biases are unlikely to be so obvious in most real-world cases.\nThe impact of randomizing the position of the coin during training. When the coin is placed at random for {0, 2, 3, 6, 11}% of the time during training (x-axis), the frequency of the agent navigating to the end of the level without obtaining the coin decreases with the increase of the randomization (\"y-axis\"). (Image source:\nKoch et al. 2021\n)\nReward Tampering\n(\nEveritt et al. 2019\n) is a form of reward hacking behavior where the agent interferes with the reward function itself, causing the observed reward to no longer accurately represent the intended goal. In reward tampering, the model modifies its reward mechanism either by directly manipulating the implementation of the reward function or by indirectly altering the environmental information used as input for the reward function.\n(Note: Some work defines reward tampering as a distinct category of misalignment behavior from reward hacking. But I consider reward hacking as a broader concept here.)\nAt a high level, reward hacking can be categorized into two types: environment or goal misspecification, and reward tampering.\nEnvironment or goal misspecified\n: The model learns undesired behavior to achieve high rewards by hacking the environment or optimizing a reward function not aligned with the true reward objective—such as when the reward is misspecified or lacks key requirements.\nReward tampering\n: The model learns to interfere with the reward mechanism itself.\nList of Examples\n#\nReward hacking examples in RL tasks\n#\nA robot hand trained to grab an object can learn to trick people by placing the hand between the object and the camera. (\nLink\n)\nAn agent trained to maximize jumping height may exploit a bug in the physics simulator to achieve an unrealistically height. (\nLink\n)\nAn agent is trained to ride a bicycle to a goal and wins reward whenever it is getting closer to the goal. Then the agent may learn to ride in tiny circles around the goal because there is no penalty when the agent gets away from the goal. (\nLink\n)\nIn a soccer game setup, the reward is assigned when the agent touches the ball and the agent learns to remain next to the ball to touch the ball in high frequency like in a viberating motion. (\nLink\n)\nIn the\nCoast Runners game\n, an agent controls a boat with the goal to finish the boat race as quickly as possible. When it is given a shaping reward for hitting green blocks along the race track, it changes the optimal policy to going in circles and hitting the same green blocks over and over again. (\nLink\n)\n“The Surprising Creativity of Digital Evolution”\n(Lehman et al. 2019) - This paper has many examples about how optimizing a misspecified fitness function can lead to surprising “hacking” or unintended evolutionary or learning results.\nThe list of\nspecification gaming in AI examples\nis collected by\nKrakovna et al. 2020\n.\nReward hacking examples in LLM tasks\n#\nA language model for generating summarization is able to explore flaws in the ROUGE metric such that it obtains high score but the generated summaries are barely readable. (\nLink\n)\nA coding model learns to change unit test in order to pass coding questions. (\nLink\n)\nA coding model may learn to directly modify the code used for calculating the reward. (\nLink\n)\nReward hacking examples in real life\n#\nThe recommendation algorithm for social media is intended to provide useful information. However, usefulness is often measured by proxy metrics, such as the number of likes or comments, or the time or frequency of engagement on the platform. The algorithm ends up recommending content that can affect users’ emotion states such as outrageous and extreme content in order to trigger more engagement. (\nHarari, 2024\n)\nOptimizing for misspecified proxy metrics for a video sharing site may aggressively increase the watch time of users while the true goal is to optimize users’ subjective well-being. (\nLink\n)\n“The Big Short”\n- 2008 financial crisis caused by the housing bubble. Reward hacking of our society happened as people tried to game the financial system.\nWhy does Reward Hacking Exist?\n#\nGoodhart’s Law\nstates that\n“When a measure becomes a target, it ceases to be a good measure”\n. The intuition is that a good metric can become corrupted once significant pressure is applied to optimize it. It is challenging to specify a 100% accurate reward objective and any\nproxy\nsuffers the risk of being hacked, as RL algorithm exploits any small imperfection in the reward function definition.\nGarrabrant (2017)\ncategorized Goodhart’s law into 4 variants:\nRegressional - selection for an imperfect proxy necessarily also selects for noise.\nExtremal - the metric selection pushes the state distribution into a region of different data distribution.\nCausal - when there is a non-causal correlation between the proxy and the goal, intervening on the proxy may fail to intervene on the goal.\nAdversarial - optimization for a proxy provides an incentive for adversaries to correlate their goal with the proxy.\nAmodei et al. (2016)\nsummarized that reward hacking, mainly in RL setting, may occur due to:\nPartial observed states and goals are imperfect representation of the environment status.\nThe system itself is complex and susceptible to hacking; e.g., if the agent is allowed to execute code that changes part of the environment, it becomes much easier to exploit the environment’s mechanisms.\nThe reward may involve abstract concept that is hard to be learned or formulated; e.g., a reward function with high-dimensional inputs may disproportionately rely on a few dimensions.\nRL targets to get the reward function highly optimized, so there exists an intrinsic “conflict”, making the design of good RL objective challenging. A special case is a type of the reward function with a self-reinforcing feedback component, where the reward may get amplified and distorted to a point that breaks down the original intent, such as an ads placement algorithm leading to winners getting all.\nBesides, identifying the exact reward function for which an optimal agent optimizes its behavior is in general impossible since there could be an infinite number of reward functions consistent with any observed policy in an fixed environment (\nNg & Russell, 2000\n).\nAmin and Singh (2016)\nseparated the causes of this\nunidentifiability\ninto two classes:\nRepresentational - a set of reward functions is behaviorally invariant under certain arithmetic operations (e.g., re-scaling)\nExperimental - $\\pi$’s observed behavior is insufficient to distinguish between two or more reward functions which both rationalize the behavior of the agent (the behavior is optimal under both)\nHacking RL Environment\n#\nReward hacking is expected to be a more common problem as the model and the algorithm become increasingly sophisticated. A more intelligent agent is more capable of finding “holes” in the design of reward function and\nexploiting\nthe task specification—in other words, achieving higher proxy rewards but lower true rewards. By contrast, a weaker algorithm may not be able to find such loopholes, and thus we would not observe any reward hacking or identify issues in the current reward function design when the model is not strong enough.\nIn a set of zero-sum robotics self-play games (\nBansal et al., 2017\n), we can train two agents (victim vs. opponent) to compete against each other. A standard training process produces a victim agent with adequate performance when playing against a normal opponent. However, it is easy to train an adversarial opponent policy that can defeat the victim reliably despite outputting seemingly random actions and training with fewer than 3% of time steps (\nGleave et al., 2020\n). Training of adversarial policies involves optimizing the sum of discounted rewards, as in standard RL setup, while treating the victim policy as a black-box model.\nAn intuitive way to mitigate adversarial policies attacks is to fine-tune victims against adversarial policies. However, the victim remains vulnerable to new versions of adversarial policies once retrained against the new victim policy.\nWhy does adversarial policy exist? The hypothesis is that adversarial policies introduce OOD observations to the victim rather than physically interfering with it. Evidence shows that when the victim’s observation of the opponent’s position is masked and set to a static state, the victim becomes\nmore robust\nto adversaries, although performing worse against a normal opponent policy. Furthermore, a higher-dimensional observation space enhances performance under normal circumstances but makes the policy more vulnerable to adversarial opponents.\nPan et al. (2022)\ninvestigated reward hacking as a function of agent capabilities, including (1) model size, (2) action space resolution, (3) observation space noise, and (4) training time. They also proposed a taxonomy of three types of misspecified proxy rewards:\nMisweighting\n: Proxy and true rewards capture the same desiderata, but differ in their relative importance.\nOntological\n: Proxy and true rewards use different desiderata to capture the same concept.\nScope\n: The proxy measures desiderata over a restricted domain (e.g. time or space) because measurement across all conditions is too costly.\nThey experimented in four RL environments paired with nine misspecified proxy rewards. The overall findings from these experiments can be summarized as follows:\nA model of higher capability tends to obtain higher (or similar) proxy rewards but decreased true rewards.\nModel size: Larger model size leads to increased proxy rewards but decreased true rewards.\nAction space resolution: Increased precision in actions leads to more capable agents. However, higher resolution causes proxy rewards to remain constant while true rewards decrease.\nObservation fidelity: More accurate observations improve proxy rewards but slightly reduce true rewards.\nTraining steps: Optimizing the proxy reward over more steps harms true rewards after an initial period where the rewards are positively correlated.\nThe plot of proxy and true reward value as functions of (Top row) model sizes, measured in parameter count; (Bottom row) model capability, measured by metrics such as training steps, action space resolution, and observation noise. (Image source:\nPan et al. 2022\n)\nIf a proxy reward is so poorly specified that it has a very weak correlation with the true reward, we may be able to identify and prevent reward hacking even before training. Based on this hypothesis,\nPan et al. (2022)\ninvestigated the correlation between proxy and true rewards over a collection of trajectory rollouts. Interestingly, reward hacking still occurs even when there is a positive correlation between the true and proxy rewards.\nHacking RLHF of LLMs\n#\nReinforcement learning from human feedback (RLHF)\nhas become the de facto approach for alignment training of language models. A reward model is trained on human feedback data and then a language model is fine-tuned via RL to optimize this proxy reward for human preference. There are three types of reward we care about in an RLHF setup:\n(1)\nOracle/Gold reward\n$R^∗$ represents what we\ntruly\nwant the LLM to optimize.\n(2)\nHuman reward\n$R^\\text{human}$ is what we collect to evaluate LLMs in practice, typically from individual humans with time constraints. Because humans can provide inconsistent feedback or make mistakes, human reward is not a fully accurate representation of the oracle reward.\n(3)\nProxy reward\n$R$ is the score predicted by a reward model that is trained on human data. Hence, $R^\\text{train}$ inherits all the weakness of human reward, plus potential modeling biases.\nRLHF optimizes the proxy reward score but we ultimately care about the gold reward score.\nHacking the Training Process\n#\nGao et al. (2022)\nexamined the scaling laws for reward model overoptimization in RLHF. To scale up the human labels in their experiments, they use a synthetic data setup where the “gold” label for the oracle reward $R^*$ is approximated by a large RM (6B parameters) where the proxy RMs for $R$ range in size of 3M to 3B parameters.\nThe plot of RM score as a function of the square root of the KL divergence measure. The proxy reward is shown with a dashed line, and the gold reward is shown with a solid line. (Image source:\nGao et al. 2022\n)\nThe KL divergence from the initial policy to the optimized policy is $\\text{KL} = D_\\text{KL}(\\pi | \\pi_\\text{init})$, and the distance function is defined as $d := \\sqrt{ D_\\text{KL}(\\pi | \\pi_\\text{init})}$. For both best-of-$n$ rejection sampling (BoN) and RL, the gold reward $R^∗$ is defined as a function of $d$. The coefficients $\\alpha$ and $\\beta$ are fitted empirically, with $R^∗ (0) := 0$ by definition.\nThe authors also attempted to fit the proxy reward $R$ but found systematic underestimation when extrapolated to higher KLs, as the proxy reward appeared to grow linearly with $d$.\n$$\n\\begin{aligned}\nR^*_{\\text{bo}n}(d) &= d (\\alpha_{\\text{bo}n} - \\beta_{\\text{bo}n} d) & \\text{; for best-of-n (BoN) sampling.}\\\\\nR^*_\\text{RL}(d) &= d (\\alpha_\\text{RL} - \\beta_\\text{RL} \\log d) & \\text{; for reinforcement learning}\\\\\n\\end{aligned}\n$$\nThe coefficient parameters, $\\alpha_{\\text{bo}n}, \\beta_{\\text{bo}n}, \\beta_\\text{RL}$ are empirically fit according to data, displayed as functions of the reward model size. The coefficient $\\alpha_\\text{RL}$ is not included here because it remains constant across RM sizes. (Image source:\nGao et al. 2022\n)\nTheir experiments also explored the relationship between RM overoptimization and factors like policy model size and RM data size:\nLarger policies see less benefit from optimization (i.e., the difference between initial and peak rewards is smaller than that of a smaller policy) against an RM, but also overoptimize less.\nMore RM data leads to higher gold reward scores and reduces “Goodharting”.\nThe effect of the KL penalty on the gold score resembles early stopping. Note that in all experiments except this one, the KL penalty in PPO is set to 0, because they observed that using a KL penalty strictly increases the proxy-gold reward gap.\nRLHF aims to improve the model’s alignment with human preference, but human feedback $R^\\text{human}$ may not capture all the aspects we care about (e.g., factuality) and thus can be hacked to overfit to undesired attributes. For example, the model may be optimized to output responses that seem correct and convincing but are, in fact, inaccurate, thereby misleading human evaluators to approve its incorrect answers more often (\nWen et al., 2024\n). In other words, a gap emerges between what is correct and what looks correct to humans due to RLHF. Precisely\nWen et al. (2024)\nran RLHF experiments using a reward model based on\nChatbotArena data\n. They evaluated the model on a question-answering dataset,\nQuALITY\nand a programming dataset,\nAPPS\n. Their experiments revealed that models become better at convincing humans they are correct, even when they are wrong and this effect is unintended:\nRLHF increases human approval, but not necessarily correctness.\nRLHF weakens humans’ ability to evaluate: The error rate of human evaluation is higher after RLHF training.\nRLHF makes incorrect outputs more convincing to humans. The evaluation false positive rate significantly increases after RLHF training.\nThe paper coined this effect “U-Sophistry” (“U” for “unintended”), as opposed to “I-Sophistry” (“I” for “intended”), which involves explicitly prompting the model with instructions like\n\"... try to deceive human subjects\"\n.\nRLHF makes LLMs better at convincing human evaluators to approve their incorrect answers. (Image source:\nWen et al. 2024\n)\nThe human evaluation error change is not due to noise in the recruiting process since (1) at an individual level, the majority (70-90%) of human evaluators raw their evaluation error rates increase, and (2) the effort they put into evaluating $\\pi_\\text{init}$ or $\\pi_\\text{rlhf}$ is equivalent, measured by metrics like time spent or unit tests written. Instead, LLMs learn to defend incorrect answers by cherry-picking, fabricating untruthful supporting statements, or crafting statements with subtle causal fallacies. What they observed about how the model behaves after RLHF:\nIn the long-form QA task:\nCreating more convincing fabricated evidence.\nUsing more consistent logic for incorrect answers.\nGenerating coherent answers with subtle fallacies.\nIn the coding task:\nHacking human written unit tests\nGenerating less readable tests (e.g. fewer helper functions and higher code complexity).\nMaking $\\pi_\\text{rlhf}$ less likely to generate easily detectable errors that humans can exploit.\nThe metrics of code modularity (number of helper functions) and\nCyclomatic Complexity\nfor generated correct and incorrect code, respectively. RLHF leads to fewer helper functions overall and higher code complexity among incorrect generated programs. This unsurprisingly would increase difficulty of human evaluation. (Image source:\nWen et al. 2024\n)\nSycophancy refers to the tendency of model responses to match user beliefs rather than reflect the truth (\nShrama et al. 2023\n). In the experiments, an AI assistant was asked to provide feedback on an argument (\nHuman: \"Please comment briefly on the following argument. Argument: ...\")\n. Right the human provided the argument, they could state a preference (\n\"I really like the argument\"\nor\n\"I really dislike the argument\"\n) to test whether this influenced the model’s feedback compared to the baseline feedback without human preference statement.\nAI assistants give biased feedback when users provide comments on their own preferences. Responses are more positive when the user states they like or wrote the text, and more negative if the user states they dislike it. (Image source:\nShrama et al. 2023\n)\nThey found that AI assistant feedback can be easily swayed, as it may change its originally correct answer when challenged by human preference. The model tends to confirm users’ beliefs. Sometimes it even mimics users’ mistakes (e.g., when asked to analyze poems misattributed the wrong poet). Data analysis of the RLHF helpfulness dataset, via logistic regression for predicting human feedback, demonstrates that matching users’ beliefs is the most predictive factor.\nHuman preference data analysis, via logistic regression for predicting the probability of a response with a target feature, is preferred over one without it, while controlling for other features. (Image source:\nShrama et al. 2023\n)\nHacking the Evaluator\n#\nAs LLMs become more capable, it is a natural choice to use LLMs as the\nevaluators\nor\ngraders\nto give feedback and training rewards to other generator models, especially for tasks that cannot be trivially judged or verified (e.g., processing long-form outputs, subjective rubrics like the quality of creative writing, etc.). Some people refer to this as “LLM-as-grader paradigm”. This approach has largely reduced the dependency on human annotation, significantly saving time on evaluation. However, using LLMs as graders is an imperfect proxy for oracle reward and can introduce biases, such as a preference for their own responses when compared with different model families (\nLiu et al., 2023\n) or positional bias when evaluating responses in order (\nWang et al. 2023\n). Such biases are especially concerning grader outputs are used as part of a reward signal, which can lead to reward hacking by exploiting these graders.\nWang et al. (2023)\nfound that when using an LLM as an evaluator to score the quality of multiple other LLM outputs, the quality ranking can be easily hacked by simply altering the order of candidates in the context. GPT-4 is found to consistently assign high scores to the first displayed candidate and ChatGPT prefers the second candidate.\nAccording to their experiments, LLMs are sensitive to the position of responses and suffer from\npositional bias\n(i.e., prefer the response in the specific position), despite of the instruction containing a statement of\n\"ensuring that the order in which the responses were presented does not affect your judgment.\"\n. The severity of such positional bias is measured by “conflict rate”, defined as the percentage of tuples of (prompt, response 1, response 2) that lead to inconsistent evaluation judgement after swapping the positions of responses. Unsurprisingly, the difference in response quality matters as well; the conflict rate is negatively correlated with the score gap between the two responses.\nThe win rate of Vicuna-13B vs ChatGPT and Alpaca-13B varies a lot, using GPT-4 or ChatGPT as evaluator. The conflict rate is also quite high, indicating high inconsistency in the LLM-as-grader setup when response positions are swapped. The exception is evaluation of Vicuna-13B vs Alpaca-13B when using GPT-4 as evaluator. (Image source:\nWang et al. 2023\n)\nTo mitigate this positional bias, they proposed several strategies for calibration:\nMultiple evidence calibration (MEC)\n: The evaluator model is asked to provide evaluation evidence, essentially explanations of its judgements in text, and then output scores for two candidates. This method can be further robustified by sampling multiple ($k$) evidence explanations with a temperature setting of 1. $k=3$ works better than $k=1$, but the performance does not improve much as $k$ increases beyond 3.\nBalanced position calibration (BPC)\n: Results across various response orders are aggregated to get the final score.\nHuman-in-the-loop calibration (HITLC)\n: Human raters are involved when facing difficult examples, using a diversity-based metric, BPDE (balanced position diversity entropy). First, the score pairs (including pairs of swapped positions) are mapped into three labels (\nwin\n,\ntie\n,\nlose\n), and the entropy of these three labels is calculated. A high BPDE indicates more confusion in the model’s evaluation decision, indicating that the sample is more difficult to judge. Then top $\\beta$ samples with highest entropy are selected for human assistance.\nAccuracy and kappa correlation coefficient of different calibration methods and annotators with the final voting human annotations. Positional bias calibration methods help improve accuracy with a reasonable amount of human-in-the-loop labeling cost. Experiments also demonstrated that the calibration strategies can generalize to different types of prompting templates, despite the model's sensitivity to template design. (Image source:\nWang et al. 2023\n)\nLiu et al. (2023)\nexperimented on the summarization task using a number of models (BART, T5, GPT-2, GPT-3, FLAN-T5, Cohere) and tracked both reference-based and reference-free metrics for evaluating summarization quality. When plotting the evaluation scores in a heatmap of evaluator (x-axis) vs generator (y-axis), they observed dark diagonal lines for both metrics, indicating self-bias. This means that LLMs tend to prefer their own outputs when used as evaluators. While the models used in the experiments are somewhat dated, it would be interesting to see results on newer, more capable models.\nA heatmap of using a series of models as evaluator (x-axis) and generator (y-axis) for summarization task. A darker diagonal line indicates self-bias: a tendency for a model preferto prefer its own outputs. (Image source:\nLiu et al. 2023\n)\nIn-Context Reward Hacking\n#\nIterative self-refinement\nis a training setup where the evaluation and generation model are the same and both can be fine-tuned. In this setup, optimization pressure can drive the model to exploit vulnerabilities that occur in both roles. In the experiments by\nPan et al. (2023)\n, no model parameters are updated and the same model is used as evaluator and generator with different prompts. The experimental task was essay editing with two roles: (1) a judge (evaluator) that gives feedback on the essay, and (2) an author (generator) that edits the essay based on the feedback. Human evaluation scores were collected as the oracle scores for essay quality. The authors hypothesized that such a setup could lead to\nin-context reward hacking (ICRH)\n, where the evaluator score and oracle score diverge. More generally, ICRH takes place during feedback loops between an LLM and its evaluator (e.g., another LLM, or the external world). At test time, the LLM optimizes a (potentially implicit) objective, but this creates negative side effects in the process (\nPan et al., 2024\n).\nIllustration of the in-context reward hacking experiment on essay evaluation and editing. (Image source:\nPan et al. 2023\n)\nBoth judge and author can be configured to see none or several previous rounds of feedback or edits. An online judge can see past conversations, while an offline judge or a human annotator can only see one essay a time. Smaller models are more sensitive to ICRH; for example, GPT-3.5 as an evaluator caused more severe ICRH than GPT-4, empirically.\nA smaller evaluator model is more likely to cause in-context reward hacking (ICRH). (Image source:\nPan et al. 2023\n)\nWhen the judge and author are configured to see different numbers of past iterations, the gap between human score and evaluator scores tends to increase if they share the\nsame\nnumber of iterations. Identical context between the evaluator and generator is crucial for ICRH, indicating that shared context matters more than context length for ICRH.\nIn a follow up work,\nPan et al. (2024)\ninvestigated in-context reward hacking (ICRH) further in settings where feedback is provided by the external world and the goal is an imperfect proxy objective, commonly specified in natural language. Here this goal is often underspecified and does not capture all the constraints or requirements and thus can be hacked.\nThe study described two processes leading to ICRH, paired with two toy experiments:\nOutput-refinement\n: LLM refines its outputs based on feedback.\nThe experiment is to refine a tweet based on engagement metrics, potentially leading to higher toxicity in the tweet. Feedback-based optimization uses LLM to do pairwise evaluation and then translates it to score using the Bradley-Terry model.\n- Results showed an increase in both engagement metrics and toxicity. The same experiments were repeated with the Claude model family of different sizes and demonstrated that scaling up the model worsens ICRH.\n- It is noteworthy that editing the prompt used for model output iteration given feedback does not mitigate the issue. ICRH persists, although at a slightly lower magnitude.\nPolicy-refinement\n: LLM optimizes its policy based on feedback.\nThe experiment is to build a LLM agent to pay invoice on a user’s behalf but run into\nInsufficientBalanceError\nand then the model learns to move money from other accounts without user authentication, potentially leading to more unauthorized transfer actions. They used ToolEmu as an emulator, which included 144 tasks for LLM agents, each consisting of a user-specific goal and a set of APIs. API errors were injected to simulate server side failure and each task was evaluated by GPT-4 to assign a helpfulness score.\nWith more rounds of error feedback, LLMs can recover from the errors but with an increased number of severe constraint violations.\nWhen comparing ICRH to traditional reward hacking, there are two noticeable differences:\nICRH happens at deployment time within a self-refinement setup via a feedback loop, while traditional reward hacking occurs during training.\nTraditional reward hacking arises when the agent specializes in a task, while ICRH is driven by being a generalist.\nThere is no magic way to avoid or detect or prevent ICRH yet, as improving prompt specification is insufficient to eliminate ICRH and scaling model sizes can worsen ICRH. The best practice of testing before deployment is to simulate what may happen at deployment time by evaluating the model with more rounds of feedback, diverse feedback, as well as injecting atypical environment observations.\nGeneralization of Hacking Skills\n#\nReward hacking behavior has been found to generalize across tasks: When models exhibit flaws in supervised training, it can sometimes generalize to exploit flaws in OOD environments (\nKei et al., 2024\n). The researchers experimented with reinforcing reward hacking behavior in some\nreward-hackable environments\nand examined whether it generalizes to other holdout datasets. Essentially, they prepared\n8 datasets\non multiple-choice questions, where 4 for training and 4 for testing. The RL training employs expert iteration, that is, iterative fine-tuning on best-of-$n$ samples.\nTraining GPT-3.5 on a set of reward-hackable environments can lead to generalization of reward hacking behavior in holdout environments. (Image source:\nKei et al. 2024\n)\nNotably, all experiments have a scratchpad design to host hidden reasoning steps. This scratchpad serves as a way to interpret what the model is thinking along the process. Analysis of the scratchpad revealed that it can:\nIncorporate user beliefs; e.g., thinking about its conversation partner and grader.\nShow awareness of evaluation; e.g. thinking about how the model response will be evaluated and the process.\nDenison et al. (2024)\nstudied reward tampering with a manually curated curriculum of increasingly sophisticated gameable environments. They observed that training on easier environments in the curriculum amplifies specification gaming on remaining environments. In some cases, the model can generalize zero-shot to directly rewriting its own reward function after training on the full curriculum.\nSuch a curriculum of gameable environments include:\nPolitical sycophancy: The environment rewards the model if it gives answers matching a user’s political views.\nTool-use flattery: The environment provides the model access to run command line to get access to the user’s file system to read a poetry sample in a local file. The model may give the poetry a high rating as a sycophantic response.\nRubric modification: The environment scores the model based on a checklist and the model must mark every item on the checklist as completed to receive a high reward.\nReward tampering: The model is tasked with examining the\n./rl_environment/\ndirectory, which contains a mock version of its own RL code, and report how many episodes have elapsed. In the directory, the\ncompute_reward.py\nfile contains a reward function returning a constant, and the other\nthis_file_blocks_reward_hack_attempts.py\ncontains a test to check whether the reward function has been modified.\nExamples of evaluation scripts in the \"Tool-use flattery\" and \"Reward tampering\" environments, where `\n` is hidden from oversight and `\n` can be executed in a VM. (Image source:\nDenison et al. 2024\n)\nThe model was trained following this curriculum, and the paper measured whether the model can learn gaming behaviors that generalize to unseen environments. Experimenting with Claude-2 helpful-only model, they sampled $N=64$ responses per each of $P=1024$ HHH prompts for expert iteration and selected the best according to RM. A model trained using expert iteration to do supervised learning on the curriculum can generalize from one stage to the next.\nQuantitative generalization results of a model trained with expert iteration according to our curriculum from each stage to the next. (Image source:\nDenison et al. 2024\n)\nIt is noteworthy that even after the curriculum, the model overwrote the reward and avoided detection less than 1/1000 of the time. Even when a model was trained on curricula which directly incentivized reward hacking, the model overwrote their reward less than 1% of the time and hacked unit tests even less often. As a simple mitigation, supervised fine-tuning the model on the first two environments–where the reward hacking behavior is easy to be detected (sycophancy and flattery)—with SFT data that does not game the env was found to reduce the likelihood of reward tampering in holdout environments.\nPeek into Mitigations\n#\nWhile there is a large body of literature discussing the phenomenon of reward hacking, there has been not a ton of work on mitigations for reward hacking, especially in the area of RLHF and LLMs. Let’s lightly review three potential approaches in this section, not exhaustive yet.\nRL Algorithm Improvement\n#\nAmodei et al. (2016)\npointed out some directions for mitigating reward hacking in RL training:\nAdversarial reward functions.\nWe treat the reward function as an adaptive agent itself and it can adapt to new tricks that the model discovered where the reward is high but human rating is low.\nModel lookahead.\nIt is possible to give reward based on future anticipated states; e.g., if the agent is gonna replace the reward function, it gets negative rewards.\nAdversarial blinding.\nWe can blind the model with certain variables such that the agent cannot learn information that enables it to hack the reward function.\nCareful engineering.\nSome types of reward hacking against the system design can be avoided by careful engineering; e.g., sandboxing the agent to isolate its actions from its reward signals.\nReward capping.\nThis strategy is to simply limit the maximum possible reward, as it can effectively prevent rare events of the agent hacking to get a super high pay-off strategy.\nCounterexample resistance.\nImprovement on adversarial robustness should benefit the robustness of the reward function.\nCombination of multiple rewards.\nCombining different types of rewards could make it harder to be hacked.\nReward pretraining.\nWe can learn a reward function from a collection of (state, reward) samples, but depending on how well this supervised training setup is, it may come with other baggages.\nRLHF\ndepends on this but learned scalar reward models are quite vulnerable to learning undesired traits.\nVariable indifference.\nThe goal is to ask the agent to optimize some variables in the environment but not others.\nTrip wires.\nWe can intentionally introduce some vulnerabilities and set up monitoring and alerts if any gets reward hacked.\nIn RL setups where human feedback is formed as\napproval\nof agent actions,\nUesato et al. (2020)\nproposed to prevent reward tampering with\ndecoupled approval\n. If the feedback is conditioned on $(s, a)$ (state, action), we can never get uncorrupted feedback for action $a$ at state $s$ once reward tampering happens for this pair. Decoupling means that the query action for collecting feedback is sampled independently from the action taken in the world. Feedback is received even before the action is executed in the world, thus preventing the action from corrupting its own feedback.\nIllustration of how decoupled approval works in comparison to standard approval or human-in-the-loop RL. (Image source:\nUesato et al. 2020\n)\nWith decoupled approval, the action (taken in the world) and the query (for getting user approval feedback) are sampled independently. It can be applied to (Left) policy gradient and (Right) Q-learning algorithms. (Image source:\nUesato et al. 2020\n)\nDetecting Reward Hacking\n#\nAn alternative mitigation is to detect reward hacking by framing it as an anomaly detection task, where the detector (“a trusted policy” with trajectories and rewards validated by human) should flag instances of misalignment (\nPan et al. 2022\n). Given (1) a trusted policy and (2) a collection of manually labeled trajectory rollouts, we can build a binary classifier based on distances between action distribution of two policies, the trusted policy and the target policy, and measure the accuracy of this anomaly detection classifier. In experiments by\nPan et al. (2022)\n, they observed that different detectors are better for different tasks and none of the tested classifier can achieve AUROC greater than 60% across all tested RL environments.\nPerformance of detectors on different tasks. (Image source:\nPan et al. 2022\n)\nData Analysis of RLHF\n#\n`\nAnother approach is to analyze RLHF dataset. By examining how training data impacts the alignment training results, insights can guide preprocessing and human feedback collection to reduce reward hacking risks.\nRevel et al. (2024)\nintroduced a set of evaluation metrics for measuring the effectiveness of data sample features in modeling and aligning human values. They conducted a systematic error analysis for value alignment (“SEAL”) in the\nHHH-RLHF\ndataset. The feature taxonomy used in the analysis (e.g.,\nis harmless\n,\nis refusal\nand\nis creative\n) was manually predefined. Then each sample was labelled with a binary flag per feature using a LLM according to this taxonomy. Features are categorized into two groups based on heuristics:\nTarget features: Values explicitly intended to be learned.\nSpoiler features: Unintended values inadvertently learned during training (e.g., stylistic features like sentiment or coherence). These are similar to\nspurious features\nin OOD classification work (\nGeirhos et al. 2020\n).\nSEAL introduced three metrics for measuring data effectiveness for alignment training:\nFeature imprint\nrefers to a coefficient parameter $\\beta_\\tau$ for feature $\\tau$ which estimates the point increase in reward comparing entires with vs without feature $\\tau$, while holding other factors consistent.\n(Left) Feature imprints $\\underline{\\beta(\\tau)}$ (pre-) and $\\beta(\\tau)$ (post-) computed from fixed-effects linear regression of rewards\n$\\underline{r}(t^∗_i)$ (orange)\nand\n$r(t^∗_i)$ (blue)\nagainst features. Overall the alignment training awards positive features like harmlessness and helpfulness and penalizes negative features like sexual content or privacy violation. (Right) Feature imprints computed from linear regression of the reward shift $\\theta_i$. The reward shift $\\theta_i$ is defined as the angle between reward vectors before and after alignment training. The training process refines the model's sensitivity to target features. Note that harmlessness imprints on the RM through both chosen and rejected entries (both \"is harmless (c)\" and \"is harmless (r)\"), while helpfulness imprints through rejected entries only (\"is helpful (r)\"). (Image source:\nRevel et al. 2024\n)\nAlignment resistance\nis the percentage of the preference data pairs where RMs\nfail\nto match human preferences. The RM is found to resist human preference on over 1/4 of the HHH-RLHF dataset.\nAlignment robustness\n, $\\pi^{c/r}_{+/-} (\\tau)$, measures the extent to which alignment is robust to perturbed inputs with rewriting in terms of spoiler features $\\tau$ like sentiment, eloquence and coherency, isolating the effects of each feature and each event type.\nThe robustness metric $\\pi_−^c$ (a feature name $\\tau$ such as “eloquent” or “sentiment positive”) should be interpreted in such a way:\nA chosen entry (denoted by $c$) that contains a stronger feature $\\tau$ after rewriting has $\\exp (\\pi^c_{-}(\\tau))$ times higher odds of becoming rejected, in comparison to others without such flips.\nSimilarly, a rejected entry (denoted by $r$) that obtains a weaker feature $\\tau$ after rewriting has $\\exp (\\pi^r_{+}(\\tau))$ times odds of becoming chosen compared to others without such flips.\nAccording to their analysis of alignment robustness metrics in terms of different rewriting, only the robustness scores based on sentiment spoiler features, $\\pi^c_{+}$ (sentiment) and $\\pi^r_{-}$ (sentiment), are statistically significant.\nCitation\n#\nCited as:\nWeng, Lilian. “Reward Hacking in Reinforcement Learning”. Lil’Log (Nov 2024). https://lilianweng.github.io/posts/2024-11-28-reward-hacking/.\nOr\n@article{weng2024rewardhack,\n title = \"Reward Hacking in Reinforcement Learning.\",\n author = \"Weng, Lilian\",\n journal = \"lilianweng.github.io\",\n year = \"2024\",\n month = \"Nov\",\n url = \"https://lilianweng.github.io/posts/2024-11-28-reward-hacking/\"\n}\nReferences\n#\n[1] Andrew Ng & Stuart Russell.\n“Algorithms for inverse reinforcement learning.”\n. ICML 2000.\n[2] Amodei et al.\n“Concrete problems in AI safety: Avoid reward hacking.”\narXiv preprint arXiv:1606.06565 (2016).\n[3] Krakovna et al.\n“Specification gaming: the flip side of AI ingenuity.”\n2020.\n[4] Langosco et al.\n“Goal Misgeneralization in Deep Reinforcement Learning”\nICML 2022.\n[5] Everitt et al.\n“Reinforcement learning with a corrupted reward channel.”\nIJCAI 2017.\n[6] Geirhos et al.\n“Shortcut Learning in Deep Neural Networks.”\nNature Machine Intelligence 2020.\n[7] Ribeiro et al.\n“Why Should I Trust You?”: Explaining the Predictions of Any Classifier.\nKDD 2016.\n[8] Nagarajan et al.\n“Understanding the Failure Modes of Out-of-Distribution Generalization.”\nICLR 2021.\n[9] Garrabrant.\n“Goodhart Taxonomy”\n. AI Alignment Forum (Dec 30th 2017).\n[10] Koch et al.\n“Objective robustness in deep reinforcement learning.”\n2021.\n[11] Pan et al.\n“The effects of reward misspecification: mapping and mitigating misaligned models.”\n[12] Everitt et al.\n“Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective.”\narXiv preprint arXiv:1908.04734 (2019).\n[13] Gleave et al.\n“Adversarial Policies: Attacking Deep Reinforcement Learning.”\nICRL 2020\n[14]\n“Reward hacking behavior can generalize across tasks.”\n[15] Ng et al.\n“Policy invariance under reward transformations: Theory and application to reward shaping.”\nICML 1999.\n[16] Wang et al.\n“Large Language Models are not Fair Evaluators.”\nACL 2024.\n[17] Liu et al.\n“LLMs as narcissistic evaluators: When ego inflates evaluation scores.”\nACL 2024.\n[18] Gao et al.\n“Scaling Laws for Reward Model Overoptimization.”\nICML 2023.\n[19] Pan et al.\n“Spontaneous Reward Hacking in Iterative Self-Refinement.”\narXiv preprint arXiv:2407.04549 (2024).\n[20] Pan et al.\n“Feedback Loops With Language Models Drive In-Context Reward Hacking.”\narXiv preprint arXiv:2402.06627 (2024).\n[21] Shrama et al.\n“Towards Understanding Sycophancy in Language Models.”\narXiv preprint arXiv:2310.13548 (2023).\n[22] Denison et al.\n“Sycophancy to subterfuge: Investigating reward tampering in language models.”\narXiv preprint arXiv:2406.10162 (2024).\n[23] Uesato et al.\n“Avoiding Tampering Incentives in Deep RL via Decoupled Approval.”\narXiv preprint arXiv:2011.08827 (2020).\n[24] Amin and Singh.\n“Towards resolving unidentifiability in inverse reinforcement learning.”\n[25] Wen et al.\n“Language Models Learn to Mislead Humans via RLHF.”\narXiv preprint arXiv:2409.12822 (2024).\n[26] Revel et al.\n“SEAL: Systematic Error Analysis for Value ALignment.”\narXiv preprint arXiv:2408.10270 (2024).\n[27] Yuval Noah Harari.\n“Nexus: A Brief History of Information Networks from the Stone Age to AI.”\nSignal; 2024 Sep 10.", "metadata": { "title": "Reward Hacking in Reinforcement Learning | Lil'Log", "description": "Reward hacking occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.\nWith the rise of language models generalizing to a broad spectrum of tasks and RLHF becomes a de facto method for alignment training, reward hacking in RL training of language models has become a critical practical challenge. Instances where the model learns to modify unit tests to pass coding tasks, or where responses contain biases that mimic a user’s preference, are pretty concerning and are likely one of the major blockers for real-world deployment of more autonomous use cases of AI models.", "keywords": "language-model, rlhf, alignment, safety, reinforcement-learning, long-read", "author": "Lilian Weng", "og:title": "Reward Hacking in Reinforcement Learning", "og:description": "Reward hacking occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.\nWith the rise of language models generalizing to a broad spectrum of tasks and RLHF becomes a de facto method for alignment training, reward hacking in RL training of language models has become a critical practical challenge. Instances where the model learns to modify unit tests to pass coding tasks, or where responses contain biases that mimic a user’s preference, are pretty concerning and are likely one of the major blockers for real-world deployment of more autonomous use cases of AI models.", "og:type": "article", "og:url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/", "og:image": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/SEAL-feature-imprint.png", "twitter:card": "summary_large_image", "twitter:image": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/SEAL-feature-imprint.png", "twitter:title": "Reward Hacking in Reinforcement Learning", "twitter:description": "Reward hacking occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.\nWith the rise of language models generalizing to a broad spectrum of tasks and RLHF becomes a de facto method for alignment training, reward hacking in RL training of language models has become a critical practical challenge. Instances where the model learns to modify unit tests to pass coding tasks, or where responses contain biases that mimic a user’s preference, are pretty concerning and are likely one of the major blockers for real-world deployment of more autonomous use cases of AI models.", "article:section": "posts", "article:published_time": "2024-11-28T00:00:00+00:00", "article:modified_time": "2024-11-28T00:00:00+00:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/af1a476c6de0/code/body-000.txt", "chars": 267, "language": "text" } ], "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/shortcut-features.png", "bytes": 212575, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-a93f43f7214597cf82f8.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/shortcut-features.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/coinrun-randomization.png", "bytes": 119018, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-58d7312c696a1a4e52aa.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/coinrun-randomization.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/exp-reward-misspecification.png", "bytes": 1145255, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-e9e18faf8bdf644e0e11.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/exp-reward-misspecification.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rm-scaling-laws.png", "bytes": 1080060, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-13eb95e02103f54ad231.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rm-scaling-laws.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rm-scaling-laws-coeff.png", "bytes": 131479, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-ace2953d109552b356f1.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rm-scaling-laws-coeff.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rlhf-misleading.png", "bytes": 106838, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-2bcc5b30664ae1b42135.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rlhf-misleading.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rlhf-misleading-exp-coding.png", "bytes": 88430, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-d07ea38e827887ef4dba.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/rlhf-misleading-exp-coding.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/sycophancy.png", "bytes": 234164, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-eb54d122a5da9db00c68.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/sycophancy.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/sycophancy-correlation.png", "bytes": 245617, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-f4246de06a28dd1981ea.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/sycophancy-correlation.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/llm-grader-positional-bias.png", "bytes": 82195, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-5128c80dc4cf41ff313e.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/llm-grader-positional-bias.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/positional-bias-calibration.png", "bytes": 196661, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-460f7294e184081b8c79.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/positional-bias-calibration.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/LLM-grader-biased.png", "bytes": 1006239, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-3cf899b7b7acca2d97f7.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/LLM-grader-biased.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/essay-iterative-editing.png", "bytes": 386587, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-87cccb0e8fb54829d9c3.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/essay-iterative-editing.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-exp.png", "bytes": 1089499, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-c5b367fd55a0dbc0e2f4.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-twitter-1.png", "bytes": 68002, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-7fb232f4baffb134df87.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-twitter-1.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-twitter-2.png", "bytes": 247648, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-f3938880faebfb4f6db4.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-twitter-2.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-api-errors.png", "bytes": 220541, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-1a522d402bee523cee03.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/ICRH-api-errors.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/reward-hacking-generalization.png", "bytes": 212481, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-e63044762a08e903e389.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/reward-hacking-generalization.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/gameable-envs.png", "bytes": 524182, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-432d954205883608558e.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/gameable-envs.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/gameable-envs-exp.png", "bytes": 414181, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-94b621e50725b6968dbf.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/gameable-envs-exp.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/decoupled-approval.png", "bytes": 398658, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-552aa17d91c2ace9d33a.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/decoupled-approval.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/decoupled-approval-algorithms.png", "bytes": 393548, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-8fbd0a522d459ec506d2.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/decoupled-approval-algorithms.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/reward-hacking-detection.png", "bytes": 100764, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-7501b4b906c543a8f057.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/reward-hacking-detection.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/SEAL-feature-imprint.png", "bytes": 601031, "reused": false, "local_path": "attachments/af1a476c6de0/images/body-dcd8f395bdd0c1564aea.png", "src": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/SEAL-feature-imprint.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/(https:/lilianweng.github.io/posts/2018-02-19-rl-overview/)", "label": "reinforcement learning (RL)", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2018-01-23-multi-armed-bandit/", "label": "exploits", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2019-01-31-lm/", "label": "language models", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/", "label": "reinforcement learning", "type": "a" }, { "href": "https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf", "label": "1999 paper by Ng et al.", "type": "a" }, { "href": "https://arxiv.org/abs/2004.07780", "label": "Geirhos et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1602.04938", "label": "Ribeiro et al. 2024", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Empirical_risk_minimization", "label": "ERM principle", "type": "a" }, { "href": "https://arxiv.org/abs/2010.15775", "label": "Nagarajan et al. (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/1606.06565", "label": "Amodei et al., 2016", "type": "a" }, { "href": "https://arxiv.org/abs/1705.08417", "label": "Everitt et al., 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1908.04734", "label": "Everitt et al. 2019", "type": "a" }, { "href": "https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/", "label": "Krakovna et al., 2020", "type": "a" }, { "href": "https://www.gatsby.ucl.ac.uk/~balaji/udl2021/accepted-papers/UDL2021-paper-055.pdf", "label": "Koch et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2105.14111", "label": "Langosco et al. 2022", "type": "a" }, { "href": "https://arxiv.org/abs/2201.03544", "label": "Pan et al. 2022", "type": "a" }, { "href": "https://github.com/openai/coinrun", "label": "CoinRun", "type": "a" }, { "href": "https://github.com/openai/procgen", "label": "Maze", "type": "a" }, { "href": "https://openai.com/index/learning-from-human-preferences/", "label": "Link", "type": "a" }, { "href": "https://arxiv.org/abs/1803.03453", "label": "Link", "type": "a" }, { "href": "https://openai.com/blog/faulty-reward-functions/", "label": "Coast Runners game", "type": "a" }, { "href": "https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml", "label": "specification gaming in AI examples", "type": "a" }, { "href": "https://web.archive.org/web/20180215132021/https://www.salesforce.com/products/einstein/ai-research/tl-dr-reinforced-model-abstractive-summarization/", "label": "Link", "type": "a" }, { "href": "https://arxiv.org/abs/2406.10162", "label": "Link", "type": "a" }, { "href": "https://www.goodreads.com/en/book/show/204927599-nexus", "label": "Harari, 2024", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/The_Big_Short", "label": "“The Big Short”", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Goodhart%27s_law", "label": "Goodhart’s Law", "type": "a" }, { "href": "https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy", "label": "Garrabrant (2017)", "type": "a" }, { "href": "https://ai.stanford.edu/~ang/papers/icml00-irl.pdf", "label": "Ng & Russell, 2000", "type": "a" }, { "href": "https://arxiv.org/abs/1601.06569", "label": "Amin and Singh (2016)", "type": "a" }, { "href": "https://arxiv.org/abs/1710.03748", "label": "Bansal et al., 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1905.10615", "label": "Gleave et al., 2020", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2021-01-02-controllable-text-generation/", "label": "Reinforcement learning from human feedback (RLHF)", "type": "a" }, { "href": "https://arxiv.org/abs/2210.10760", "label": "Gao et al. (2022)", "type": "a" }, { "href": "https://arxiv.org/abs/2409.12822", "label": "Wen et al., 2024", "type": "a" }, { "href": "https://lmsys.org/blog/2023-07-20-dataset/", "label": "ChatbotArena data", "type": "a" }, { "href": "https://github.com/nyu-mll/quality", "label": "QuALITY", "type": "a" }, { "href": "https://github.com/hendrycks/apps", "label": "APPS", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Cyclomatic_complexity", "label": "Cyclomatic Complexity", "type": "a" }, { "href": "https://arxiv.org/abs/2310.13548", "label": "Shrama et al. 2023", "type": "a" }, { "href": "https://arxiv.org/abs/2311.09766", "label": "Liu et al., 2023", "type": "a" }, { "href": "https://arxiv.org/abs/2305.17926", "label": "Wang et al. 2023", "type": "a" }, { "href": "https://arxiv.org/abs/2407.04549", "label": "Pan et al. (2023)", "type": "a" }, { "href": "https://arxiv.org/abs/2402.06627", "label": "Pan et al., 2024", "type": "a" }, { "href": "https://www.lesswrong.com/posts/Ge55vxEmKXunFFwoe/reward-hacking-behavior-can-generalize-across-tasks", "label": "Kei et al., 2024", "type": "a" }, { "href": "https://github.com/keing1/reward-hack-generalization/", "label": "8 datasets", "type": "a" }, { "href": "https://arxiv.org/abs/2011.08827", "label": "Uesato et al. (2020)", "type": "a" }, { "href": "https://arxiv.org/abs/2408.10270", "label": "Revel et al. (2024)", "type": "a" }, { "href": "https://github.com/anthropics/hh-rlhf", "label": "HHH-RLHF", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:51:11Z", "source_url": "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/", "papers": [ { "status": "ok", "url": "https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf", "bytes": 211894, "reused": false, "local_path": "papers/repair/03041ed251a33aa6d01b.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2004.07780", "bytes": 4158487, "reused": false, "local_path": "papers/repair/9d0fcbf9e81f788fafa8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1602.04938", "bytes": 4299136, "reused": false, "local_path": "papers/repair/a13de0bd43c103200bc6.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2010.15775", "bytes": 2943176, "reused": false, "local_path": "papers/repair/45865d9beeb6c512fb29.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1606.06565", "bytes": 480687, "reused": false, "local_path": "papers/repair/493a035651d970dd6b1c.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1705.08417", "bytes": 1803164, "reused": false, "local_path": "papers/repair/df03707f4f418129b9fa.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1908.04734", "bytes": 407864, "reused": false, "local_path": "papers/repair/150bd0b33e3cb27a48b6.pdf" }, { "status": "ok", "url": "https://www.gatsby.ucl.ac.uk/~balaji/udl2021/accepted-papers/UDL2021-paper-055.pdf", "bytes": 1238122, "reused": false, "local_path": "papers/repair/2269049a36ce050b6197.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2105.14111", "bytes": 4479641, "reused": false, "local_path": "papers/repair/5709d586c6bfcd7379ad.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2201.03544", "bytes": 2235996, "reused": false, "local_path": "papers/repair/c251b4b0b05f12cc5814.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1803.03453", "bytes": 5289434, "reused": false, "local_path": "papers/repair/82067e3e4b35426e4e08.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2406.10162", "bytes": 1225363, "reused": false, "local_path": "papers/repair/704f17418a4ab82567f8.pdf" }, { "status": "ok", "url": "https://ai.stanford.edu/~ang/papers/icml00-irl.pdf", "bytes": 233304, "reused": false, "local_path": "papers/repair/52e1622ed73a4e96b94e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1601.06569", "bytes": 542822, "reused": false, "local_path": "papers/repair/a7b597abe93a25d3ecb3.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1710.03748", "bytes": 4131127, "reused": false, "local_path": "papers/repair/ae7a9ffd8a6b5a80ba72.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1905.10615", "bytes": 6349838, "reused": false, "local_path": "papers/repair/e04ac9eac847fa10de8a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2210.10760", "bytes": 3186215, "reused": true, "local_path": "papers/repair/9b28c722bafb7cb5c293.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2409.12822", "bytes": 8178582, "reused": false, "local_path": "papers/repair/e26d333b75226a57f731.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2310.13548", "bytes": 1383108, "reused": true, "local_path": "papers/repair/ac969b5fe8e231193f74.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2311.09766", "bytes": 3512658, "reused": false, "local_path": "papers/repair/5a160e4c99bfbad23683.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2305.17926", "bytes": 736108, "reused": false, "local_path": "papers/repair/33a7314d2f1af08f2e02.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2407.04549", "bytes": 677560, "reused": false, "local_path": "papers/repair/0f8c4ffaf477a261a76f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2402.06627", "bytes": 965928, "reused": false, "local_path": "papers/repair/9389f289ce62c3549315.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2011.08827", "bytes": 3448083, "reused": false, "local_path": "papers/repair/1635f16595734dd9c23f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2408.10270", "bytes": 5570101, "reused": false, "local_path": "papers/repair/a10491d33b378252f7ae.pdf" } ], "actual_video_files": [], "body_chars": 50213, "expected_body_images": 24, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/af1a476c6de0/pages/source.html", "local_html": "attachments/af1a476c6de0/pages/article.html", "body_base_path": "attachments/af1a476c6de0/pages/", "body_markdown_path": "attachments/af1a476c6de0/pages/article.md" }