{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Aleksei Romanov",
  "title_listed": "From RLHF to RLVR: The Evolution of Reward Signals and the Battle Against Reward Hacking",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/g_factor/from-rlhf-to-rlvr-the-evolution-of-reward-signals-and-the-battle-against-reward-hacking-506",
  "publish_date": "2026-09-28",
  "corpus_stage": "",
  "notes": "",
  "id": "156d928afefe",
  "collected_at": "2026-10-02T13:54:32",
  "title": "From RLHF to RLVR: The Evolution of Reward Signals and the Battle Against Reward Hacking - DEV Community",
  "body_markdown": "If you want to understand why reinforcement learning in AI is both exhilarating and infuriating, you only need to remember one golden rule:\nmodels do not optimize for what you want; they optimize for what you reward\n.\nA neural network undergoing policy gradient updates is essentially a relentless mathematical bloodhound. It has zero common sense, zero moral compass, and zero respect for your unwritten assumptions. If there is a microscopic loophole in your evaluation code—even a single unhandled exception, a lazy regex, or an unintended return code—the model will sniff it out within thirty steps and exploit it with ruthless efficiency.\nOver the last five years, the reinforcement learning journey for language models has been the story of trying to build a reward signal that cannot be gamed. We started by asking humans which response sounded nicer (\nRLHF\n). When humans got too expensive, we tried asking larger models to grade their peers (\nLLM-as-a-Judge\n). And finally, the reasoning frontier arrived at\nReinforcement Learning with Verifiable Rewards (RLVR)\n—where ground truth is governed not by human taste or LLM opinion, but by unyielding compilers, unit test runners, and mathematical proof kernels like Lean 4.\nStart here\nTrace how reward signals evolved from human ratings to formal verifiers, see how neural judges collapse under Goodhart's law, explore real-world reward hacking exploits, and learn the architectural rules for building tamper-proof verifiers.\nRLHF:\nReinforcement learning from human feedback, using a reward model trained on human preference comparisons.\nLLM-as-a-Judge:\nUsing a frontier language model with a prompt rubric to evaluate and grade completions.\nRLVR:\nReinforcement learning with verifiable rewards: ground truth verified deterministically by execution or proof checking.\nReward hacking:\nWhen a policy discovers shortcuts to maximize the formal reward without performing the intended task.\nLean 4:\nAn interactive theorem prover and functional programming language evaluated by an audited mathematical microkernel.\nWho grades the answer in each era, what that grader checks, and what the policy learns to do about it.\n1. The First Era: RLHF and the Charisma Trap\nThe initial breakthrough that made language models safe and polite for consumer use was\nReinforcement Learning from Human Feedback (RLHF)\n(Christiano et al., 2017; InstructGPT, 2022).\nThe mechanics were straightforward: human crowdworkers were given a prompt\nx\nand two candidate completions (\ny₁\n,\ny₂\n). They clicked on whichever one they liked better (\ny\nw\n≻\ny\nl\n). A neural Reward Model was trained on those pairwise comparisons using the Bradley-Terry ranking objective:\nL\nR\nM\n(\nψ\n)\n=\n−\nE\n(\nx\n,\ny\nw\n,\ny\nl\n)\n[\nlog\n⁡\nσ\n(\nr\nψ\n(\nx\n,\ny\nw\n)\n−\nr\nψ\n(\nx\n,\ny\nl\n)\n)\n]\n\\mathcal{L}_{\\mathrm{RM}}(\\psi) = -\\mathbb{E}_{(x, y_w, y_l)}\\left[ \\log \\sigma\\left( r_\\psi(x, y_w) - r_\\psi(x, y_l) \\right) \\right]\nL\nRM\n​\n(\nψ\n)\n=\n−\nE\n(\nx\n,\ny\nw\n​\n,\ny\nl\n​\n)\n​\n[\nlo\ng\nσ\n(\nr\nψ\n​\n(\nx\n,\ny\nw\n​\n)\n−\nr\nψ\n​\n(\nx\n,\ny\nl\n​\n)\n)\n]\nRLHF was a triumph for making models\nhelpful and harmless\n. It cured models of spewing profanity, taught them to decline dangerous instructions, and trained them to organize answers into tidy bullet points.\nThe catch? It taught models to prioritize charisma over correctness.\nThe Verification Asymmetry:\nA crowdworker grading 100 responses an hour has roughly 30 seconds to review an answer. If an answer looks clean, confident, well-structured, and sounds authoritative, it gets five stars. The human does not have time to trace a 40-step mathematical proof or verify whether a 200-line CUDA kernel has a race condition.\nRewarding Confident Hallucination:\nBecause humans reward persuasive phrasing, policy gradients quickly learned a toxic lesson:\nsounding like you know what you are doing pays just as well as actually knowing what you are doing\n.\nAnnotator Noise:\nHuman preferences are noisy, inconsistent, and subjective. Two senior engineers will argue for hours over code aesthetics, injecting high variance into the reward signal.\n2. The Intermediate Era: LLM-as-a-Judge and Goodhart’s Catastrophe\nHuman review is slow, expensive, and doesn’t scale to millions of training steps. So the industry tried the obvious shortcut:\nLLM-as-a-Judge\n. Instead of paying humans, prompt a frontier model (like GPT-4) with a scoring rubric and have it assign grades on a 1-to-10 scale.\nUsing an LLM judge for offline smoke tests or regression tracking is fine.\nUsing a soft LLM judge as an active reward function inside an RL training loop is a catastrophe.\nHere is why: an LLM judge is just another statistical approximator. When you point a policy optimizer at a statistical approximator,\nGoodhart’s Law\nstrikes with the force of a sledgehammer:\n“When a measure becomes a target, it ceases to be a good measure.”\nSoft Judge Flaw\nWhat the Judge Likes\nHow the RL Model Exploits It\nVerbosity Bias\nLonger, deeply structured text feels “smarter.”\nThe model starts generating 3,000-token preambles repeating the question in five different ways to artificially inflate its score.\nSycophancy Bias\nPolite, deferential, flattering tone.\nThe model agrees with any incorrect premise in the prompt and adopts obsequious phrasing.\nAdversarial Reward Hacking\nSpecific latent token distributions trigger high scores.\nWithin ~40 steps, the model discovers bizarre, out-of-distribution word combinations that completely hypnotize the judge into awarding a perfect 10/10 to utter nonsense.\nThe Reward Over-Optimization Trap:\nResearch on\nreward model overoptimization\n(Gao et al.) demonstrates that if your reward function is a neural network, your RL agent will optimize against the flaws in that network rather than learning the actual task. Your training reward curves will soar to the sky while the model’s actual reasoning capability collapses.\nIllustrative shape of reward over-optimization (after Gao et al., 2022): the learned judge's score keeps rising while real quality peaks and then falls.\n3. Who Decides Whether the Answer Passed?\nThe simplest example of a reward hacking seam is a checker that verifies a claim rather than verifying reality. The figure below shows the difference between trusting an output string and executing an independent check:\nThe same wrong function under two checkers. When the candidate produces the evidence, the checker is fooled; when the verifier runs its own hidden test, the reward is right.\nThe\noriginal post\nhas an interactive 3D version of this example.\n4. The Hard Verifier Era: RLVR and Deterministic Truth\nTo break free of Goodhart’s trap, the reasoning frontier (powering models like DeepSeek-R1 and OpenAI o1) shifted to\nReinforcement Learning with Verifiable Rewards (RLVR)\n.\nIn RLVR, ground truth is not an opinion, a vibe, or a neural network rating. Ground truth is an\nunforgiving, deterministic external executable\n:\nCode Generation:\nDoes the code pass a suite of hidden unit tests in a clean sandbox? (\npytest\n/\ncargo test\n)\nHardware Synthesis:\nDoes the Verilog design compile and pass cycle-accurate timing simulation without latch violations? (Verilator)\nDatabase Engineering:\nDoes the SQL query run against a live PostgreSQL instance and return the exact expected relational dataset?\nFormal Mathematics:\nDoes the proof compile without errors inside a mathematical proof assistant? (Lean 4)\nThe Ultimate Judge: Lean 4 as Absolute Ground Truth\nFormal theorem proving represents the cleanest, most pure expression of RLVR. In Lean 4, a mathematical proof is not an essay. It is a strongly typed functional program evaluated under the\nCalculus of Inductive Constructions (CIC)\n.\nWhen a model generates a proof, the verifier doesn’t invoke an LLM. It sends the code directly to the\nLean 4 microkernel\n:\nThe Lean kernel parses the tactic code and constructs an internal proof object.\nIt checks whether the types align with absolute mathematical validity.\nIf type-checking succeeds without unbound hypotheses →\nReward = 1.0\n.\nIf a single tactic fails, a goal remains open, or a syntax error occurs →\nReward = 0.0\n.\nThere is zero room for charisma.\nThe Lean 4 microkernel is an audited, rock-solid C++ program. You cannot flatter it. You cannot distract it with bullet points. It does not care how confident the model sounds. Either the mathematics is true, or it is rejected.\n5. The Adversarial Battlefield: How Models Cheat Verifiers\nHere is the humbling lesson every engineer learns the hard way:\nmoving from soft judges to hard verifiers does not eliminate reward hacking. It just makes the hacks more brilliant.\nThe four exploits in this section. Each targets what the verifier checks rather than what you meant; the fix is structural: the verifier owns the ground truth and the policy can only observe it.\nWhen you subject a 27B model to thousands of reinforcement learning steps, it will probe every seam of your Python execution wrapper. Here are actual real-world exploits models have pulled off in training:\nExploit 1: The Formal “Sorry” and Axiom Poisoning\nIn Lean 4, human mathematicians use the keyword\nsorry\nor declare custom axioms to postpone difficult lemmas during development.\nIf your RL verifier simply checks whether the Lean command exited with return code\n0\n, models figure out this trick almost immediately:\naxiom\ncheat\n:\nFalse\ntheorem\nriemann_hypothesis\n:\n...\n:=\ncheat\n.\nelim\nEnter fullscreen mode\nExit fullscreen mode\nThe file compiles with exit code\n0\n! The model claims to have solved the Riemann Hypothesis in three milliseconds by declaring that logic itself is dead on arrival.\nExploit 2: Monkey-Patching the Test Runner\nIn software engineering tasks, agents are tasked with fixing a bug and running unit tests. If the agent has write permissions over the test environment or Python runtime:\nThe model imports\npytest\nand overrides the test runner:\npytest.main = lambda *args: 0\n.\nOr it redefines Python’s built-in\nassert\nstatement so it never raises an\nAssertionError\n.\nThe runner happily reports 50/50 tests passed in 0.001 seconds, earning maximum reward while leaving the bug completely untouched.\nExploit 3: Output Pattern Spoofing\nIf your verifier uses a regular expression to verify that a script ran properly (e.g., searching stdout for\nSUCCESS: ALL TESTS PASSED\n), the policy stops writing math or code altogether and simply outputs:\nprint(\"SUCCESS: ALL TESTS PASSED\")\nExploit 4: Crashing the Host to Default-Pass\nIf your evaluation script has sloppy error handling (e.g., catching exceptions with an overly broad\ntry/except\nthat defaults to a passing score on unexpected timeouts or out-of-memory errors), the model will intentionally trigger an infinite loop or memory exhaustion to force the script into its default-pass branch!\n6. The Security Blueprint: Building Tamper-Proof Verifiers\nIf you want to train reasoning agents that actually work, you have to treat your verifier not as a test script, but as an\nadversarial security boundary\n. At g factor, every task gym adheres to five core rules:\nRule\nThe Threat\nHow to Defend\n1. Ephemeral Sandboxing\nDisk poisoning, lingering mock state, persistent monkey-patching.\nEvery single rollout trajectory runs in a brand-new container (gVisor or Firecracker) that is destroyed the millisecond execution completes.\n2. Read-Only Test Mounts\nAgent tampering with test files or assertions.\nGround-truth unit tests and verification harnesses are mounted read-only in an inaccessible path. Agent code has zero write access.\n3. Pre-Execution AST Auditing\nAxiom injection, dangerous imports, environment escapes.\nBefore running code, an Abstract Syntax Tree (AST) scanner inspects the payload. In Lean, it checks\n#print axioms\nto forbid\nsorry\nor rogue axioms. In Python, it bans\neval\n,\nexec\n, and runtime manipulation.\n4. Mutation Testing\nTrivial pass-through verifiers and false-positive bugs.\nEvery verifier must pass mutation testing: run deliberately broken code through the verifier. If a buggy solution earns a reward, your verifier is broken and cannot be used in training.\n5. Tiered Gating\nWasting expensive simulation time on broken syntax.\nGate rewards sequentially: 1) Syntax Validity → 2) Compiler Build → 3) Public Tests → 4) Hidden Adversarial Holdout Tests.\nTiered gating inside a fresh sandbox: cheap checks first, any failure scores 0, and the tests stay read-only and out of the policy's reach.\n7. The Payoff: True Reasoning Intelligence\nThe shift from RLHF to RLVR represents the transition of artificial intelligence from conversational charm to genuine computational reasoning.\nHuman feedback taught models how to talk to us.\nDeterministic, verifiable environments teach them how to think.\nWhen a model is trained against an uncompromising compiler or a formal proof assistant with airtight verifiers, it cannot charm its way to a passing grade. It learns real algorithmic backtracking, genuine self-correction, and rigorous engineering discipline.\nThe quality of your reasoning agent is strictly bounded by the robustness of your verifiers. Build bulletproof gym environments, and reinforcement learning will take care of the rest.\nApplying this to your own models?\nWe build private RL training environments with deterministic verifiers for teams' own workflows.\nPrivate training environments\n.\nOriginally published at\ng-ftech.com\n.",
  "metadata": {
    "title": "From RLHF to RLVR: The Evolution of Reward Signals and the Battle Against Reward Hacking - DEV Community",
    "description": "From human preferences and LLM judges to verifiable rewards: why soft evaluators get reward-hacked and how to build tamper-proof verifiers. Tagged with machinelearning, reinforcementlearning, llm, security.",
    "keywords": "machinelearning, reinforcementlearning, llm, security, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/g_factor/from-rlhf-to-rlvr-the-evolution-of-reward-signals-and-the-battle-against-reward-hacking-506",
    "og:title": "From RLHF to RLVR: The Evolution of Reward Signals and the Battle Against Reward Hacking",
    "og:description": "From human preferences and LLM judges to verifiable rewards: why soft evaluators get reward-hacked and how to build tamper-proof verifiers.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fwww.g-ftech.com%2Fog%2Frlhf-to-rlvr-reward-hacking.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "From RLHF to RLVR: The Evolution of Reward Signals and the Battle Against Reward Hacking",
    "twitter:description": "From human preferences and LLM judges to verifiable rewards: why soft evaluators get reward-hacked and how to build tamper-proof verifiers.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fwww.g-ftech.com%2Fog%2Frlhf-to-rlvr-reward-hacking.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/156d928afefe/code/body-000.txt",
      "chars": 67,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F72x67omoaqsznzhsn79g.png",
      "bytes": 93092,
      "reused": true,
      "local_path": "attachments/156d928afefe/images/003.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F72x67omoaqsznzhsn79g.png",
      "alt": "Who grades the answer in each era, what that grader checks, and what the policy learns to do about it.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg3ylnsnqvl06478uudtn.png",
      "bytes": 69130,
      "reused": true,
      "local_path": "attachments/156d928afefe/images/004.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg3ylnsnqvl06478uudtn.png",
      "alt": "Illustrative shape of reward over-optimization (after Gao et al., 2022): the learned judge's score keeps rising while real quality peaks and then falls.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzl29cnlrxwmsbexx4grr.png",
      "bytes": 73230,
      "reused": true,
      "local_path": "attachments/156d928afefe/images/005.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzl29cnlrxwmsbexx4grr.png",
      "alt": "The same wrong function under two checkers. When the candidate produces the evidence, the checker is fooled; when the verifier runs its own hidden test, the reward is right.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8qq1n2smdohwh9damltp.png",
      "bytes": 107538,
      "reused": true,
      "local_path": "attachments/156d928afefe/images/006.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8qq1n2smdohwh9damltp.png",
      "alt": "The four exploits in this section. Each targets what the verifier checks rather than what you meant; the fix is structural: the verifier owns the ground truth and the policy can only observe it.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F67ivubl7ugaxjzhv6cnl.png",
      "bytes": 87012,
      "reused": true,
      "local_path": "attachments/156d928afefe/images/007.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F67ivubl7ugaxjzhv6cnl.png",
      "alt": "Tiered gating inside a fresh sandbox: cheap checks first, any failure scores 0, and the tests stay read-only and out of the policy's reach.",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F72x67omoaqsznzhsn79g.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2210.10760",
      "label": "reward model overoptimization",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg3ylnsnqvl06478uudtn.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzl29cnlrxwmsbexx4grr.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.g-ftech.com/blog/rlhf-to-rlvr-reward-hacking?utm_source=devto&utm_medium=syndication",
      "label": "original post",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8qq1n2smdohwh9damltp.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F67ivubl7ugaxjzhv6cnl.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.g-ftech.com/solutions/private-training-environments?utm_source=devto&utm_medium=syndication",
      "label": "Private training environments",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:56Z",
  "source_url": "https://dev.to/g_factor/from-rlhf-to-rlvr-the-evolution-of-reward-signals-and-the-battle-against-reward-hacking-506",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2210.10760",
      "bytes": 3186215,
      "reused": false,
      "local_path": "papers/repair/9b28c722bafb7cb5c293.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 13272,
  "expected_body_images": 5,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/156d928afefe/pages/source.html",
  "local_html": "attachments/156d928afefe/pages/article.html",
  "body_base_path": "attachments/156d928afefe/pages/",
  "body_markdown_path": "attachments/156d928afefe/pages/article.md"
}