{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Philipp Schmid",
  "title_listed": "DeepSeek R1",
  "keywords": "DeepSeek R1,推理模型,开源",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://www.philschmid.de/deepseek-r1",
  "publish_date": "2025-01",
  "corpus_stage": "推理部署",
  "notes": "DeepSeek R1推理模型解析",
  "id": "7dce574d8a74",
  "collected_at": "2026-10-02T14:29:18",
  "title": "Bite: How Deepseek R1 was trained",
  "body_markdown": "Bite: How Deepseek R1 was trained\nJanuary 17, 2025\n4\nminute read\nDeepSeek AI released DeepSeek-R1, an open model that rivals OpenAI's o1 in complex reasoning tasks, introduced using Group Relative Policy Optimization (GRPO) and RL-focused multi-stage training approach.\nUnderstanding Group Relative Policy Optimization (GRPO)\nGroup Relative Policy Optimization (GRPO) is a reinforcement learning algorithm to improve the reasoning capabilities of LLMs. It was introduced in the\nDeepSeekMath\npaper in the context of mathematical reasoning. GRPO modifies the traditional Proximal Policy Optimization (PPO) by eliminating the need for a value function model. Instead, it estimates baselines from group scores, reducing memory usage and computational overhead. GRPO, now also used by the Qwen team, can be used with rule/binary-based Rewards as well as General Reward Models to improve models on helpfulness.\nSampling\n: Generate multiple outputs for each prompt using the current policy\nReward Scoring\n: Each generation is scored using a reward function, could be (rule-based or outcome-based)\nAdvantage Calculation\n: The average reward of the generated outputs is used as a baseline. The advantage of each solution within the group is then computed relative to this baseline. The reward is normalized within a group.\nPolicy Optimization\n: The policy tries to maximize the GRPO objective, which includes the calculated advantages and a KL divergence term. This is different from how PPO implements the KL term within the reward.\nThe Key Differences from Proximal Policy Optimization (PPO) are\nNo Value Function\n: Unlike PPO, GRPO does not rely on a separate value function model, which simplifies training and reduces memory consumption.\nGroup-Based Advantage\n: GRPO uses the average reward of a group of outputs as a baseline. This approach better aligns with the nature of reward model training, which often examines multiple outputs for one single input.\nKL Divergence:\nGRPO directly incorporates the KL divergence term into the loss function, while PPO often uses it as part of the reward signal.\nExhibit: Pure Reinforcement Learning (R1-zero)\nIn building DeepSeek R1, the team gained deep insights from experimenting with reinforcement learning on their base model. Starting with DeepSeek V3, they applied GRPO to unsupervised reasoning text completions rule-based reward models that focused on aspects like format, mathematics, and coding:\nAccuracy rewards\n: Evaluate whether the response is correct, correct result or compiled LeetCode problem.\nFormat rewards\n: Evaluate the format that enforces the model to put its thinking process between ‘\n’ and ‘\n’ tags.\nThis leads to a pass@1 score on AIME 2024 increasing from 15.6% to 71.0%, reaching performance levels comparable to OpenAI-o1-0912 alongside output token length per problem increasing, indicating the model naturally learns to solve tasks with more thinking time/token generation.\nThis has the drawback of leading to poor readability and language mixing but it was solved for R1 using a multi-stage approach with alternating SFT → RL steps.\nThe Multi-Stage Training of DeepSeek R1\nTo prevent the early unstable cold start phase of reinforcement training (RL) training from the base model, the team started with supervised fine-tuning.\nStage 1/4 Base to Supervised Fine-Tuning (SFT)\nCollected up to 10k token-long chain-of-thought (CoT) using the fine-tuned models, R1-zero and human annotator. The data is used to fine-tune Deepseek V3 base to improve readbility and coherence.\nStage 2/4 RL for Reasoning\nUsed the same RL pipeline as R1-Zero, focusing on reasoning-intensive tasks such as coding and math using the same Rule-Based Reward Models. This time, an additional reward for \"language consistency\" is used to help the model stick to the same language.\nStage 3/4 Rejection Sampling and SFT\nGenerated large synthetic dataset using Reject Sampling (RS) focusing on writing, role-playing, and other general-purpose tasks. The model from Stage 2 was used with Deepseek V3 as a Judge to generate 600k reasoning-related samples and 200k for writing, role-playing, and other general-purpose tasks using portions of the SFT dataset of DeepSeek-V3 or regenerating them with CoT included.\nStage 4/4 RL for Helpfulness\nIn the Final Stage, GRPO is used again with a combination of Rule-Based and Outcome Reward Models to improve the model's helpfulness and harmlessness. Leading to the\nDeepseek R1\nmodel.\nSurprises\nDeepSeek didn't use Monte Carlo Tree Search (MCTS) or Process Reward Models (PRM).\nFine-tuning before applying GRPO can actually make the training process faster and more stable.\nRule-based rewards focused on accuracy and format are more effective than complex rewards models.",
  "metadata": {
    "title": "Bite: How Deepseek R1 was trained",
    "description": "5 Minute Read on how Deepseek R1 was trained using Group Relative Policy Optimization (GRPO) and RL-focused multi-stage training approach.",
    "keywords": null,
    "author": "Philipp Schmid",
    "og:title": "Bite: How Deepseek R1 was trained",
    "og:description": "5 Minute Read on how Deepseek R1 was trained using Group Relative Policy Optimization (GRPO) and RL-focused multi-stage training approach.",
    "og:url": "https://www.philschmid.de/deepseek-r1",
    "og:image": "https://www.philschmid.de/static/blog/deepseek-r1/thumbnail.jpg",
    "og:image:width": "1200",
    "og:image:height": "630",
    "og:image:alt": "Bite: How Deepseek R1 was trained",
    "og:type": "article",
    "twitter:card": "summary_large_image",
    "twitter:title": "Bite: How Deepseek R1 was trained",
    "twitter:description": "5 Minute Read on how Deepseek R1 was trained using Group Relative Policy Optimization (GRPO) and RL-focused multi-stage training approach.",
    "twitter:image": "https://www.philschmid.de/static/blog/deepseek-r1/thumbnail.jpg",
    "article:published_time": "2025-01-17"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/deepseek-r1/grpo.png",
      "bytes": 256274,
      "reused": false,
      "local_path": "attachments/7dce574d8a74/images/body-82cb453185bc6c1fdc17.png",
      "src": "https://www.philschmid.de/static/blog/deepseek-r1/grpo.png",
      "alt": "grpo.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/deepseek-r1/prompt.png",
      "bytes": 129625,
      "reused": false,
      "local_path": "attachments/7dce574d8a74/images/body-398bc7e1b000999cdf2b.png",
      "src": "https://www.philschmid.de/static/blog/deepseek-r1/prompt.png",
      "alt": "prompt.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/deepseek-r1/r1-zero.png",
      "bytes": 187255,
      "reused": false,
      "local_path": "attachments/7dce574d8a74/images/body-f7d6f0d435f93339aada.png",
      "src": "https://www.philschmid.de/static/blog/deepseek-r1/r1-zero.png",
      "alt": "r1-zero.png",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://www.philschmid.de/static/blog/deepseek-r1/evals.png",
      "bytes": 160085,
      "reused": false,
      "local_path": "attachments/7dce574d8a74/images/body-8c4618de30c73329ed69.png",
      "src": "https://www.philschmid.de/static/blog/deepseek-r1/evals.png",
      "alt": "evals.png",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://arxiv.org/abs/2402.03300",
      "label": "DeepSeekMath",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/deepseek-ai/DeepSeek-R1",
      "label": "Deepseek R1",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:53Z",
  "source_url": "https://www.philschmid.de/deepseek-r1",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2402.03300",
      "bytes": 1844642,
      "reused": true,
      "local_path": "papers/repair/dd0de0096b51ee9b7ed5.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 4747,
  "expected_body_images": 4,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/7dce574d8a74/pages/source.html",
  "local_html": "attachments/7dce574d8a74/pages/article.html",
  "body_base_path": "attachments/7dce574d8a74/pages/",
  "body_markdown_path": "attachments/7dce574d8a74/pages/article.md"
}