{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Aleksei Romanov",
  "title_listed": "SFT vs. RL: What Changes Inside the Model?",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://dev.to/g_factor/sft-vs-rl-what-changes-inside-the-model-30ho",
  "publish_date": "2026-09-24",
  "corpus_stage": "",
  "notes": "",
  "id": "45dff3e10a00",
  "collected_at": "2026-10-02T13:54:41",
  "title": "SFT vs. RL: What Changes Inside the Model? - DEV Community",
  "body_markdown": "In modern generative AI post-training, two fundamental paradigms dominate:\nSupervised Fine-Tuning (SFT)\nand\nReinforcement Learning with Verifiable Rewards (RLVR / GRPO)\n.\nWhile practitioners often treat SFT and RL as interchangeable steps on an incremental tuning ladder, they perform mathematically and mechanistically distinct operations on a model’s internal parameters:\nSFT is Knowledge Injection:\nIt uploads new factual domain knowledge, API syntax, vocabulary associations, and conversational formatting into the model’s representation space.\nRL is Reasoning, Search & Routing:\nIt does\nnot\ninject new facts. Instead, it teaches the model how to systematically explore, evaluate, backtrack, and route its existing pre-trained capabilities to solve verifiable multi-step problems.\nRecent breakthrough work by Zhu et al. (July 2026,\narXiv:2607.19331,\n“ISO: An RLVR-Native Optimization Stack”\n) provides rigorous mathematical proof of this dichotomy through the lens of\nSingular Value Decomposition (SVD)\n: during RLVR post-training, the\nsingular value spectrum (Σ) of transformer weight matrices remains virtually unchanged\nfrom the base model. RL adaptation occurs almost entirely through rotations of the\nsingular coordinate frames (\nU\nand\nV\n)\n.\nStart here\nExplore how a weight matrix is decomposed into three matrices (U, Σ, Vᵀ), why SFT rewrites the diagonal spectrum to upload new facts, and why RLVR freezes the spectrum while rotating coordinate frames to master multi-step reasoning.\nSVD Decomposition:\nFactoring a weight matrix W into U · Σ · Vᵀ (output frame, singular values spectrum, input frame).\nSpectrum (Σ):\nA diagonal matrix of singular values representing the energy and capacity allocated to concepts in the model.\nSingular Frames (U, V):\nOrthonormal coordinate systems that detect incoming features (Vᵀ) and route outputs to the next layer (U).\nSpectral Inheritance:\nThe empirical discovery that reinforcement learning preserves the base model's singular spectrum (Σ ≈ Σ₀).\nSFT grows the singular values Σ to store new facts. RLVR keeps Σ ≈ Σ₀ and rotates the frames U and Vᵀ instead.\nThe\noriginal post\nhas an interactive 3D version of this diagram.\n1. The Geometry of a Weight Matrix: SVD Decomposition\nAny weight projection tensor in a transformer layer—whether in self-attention (\nW\nq\n,\nW\nk\n,\nW\nv\n,\nW\no\n) or feed-forward MLP projections (\nW\ngate\n,\nW\nup\n,\nW\ndown\n)—can be factored into three fundamental linear algebraic components via Singular Value Decomposition:\nW\n=\nU\n⋅\nΣ\n⋅\nV\nT\nW = U \\cdot \\Sigma \\cdot V^{\\mathsf T}\nW\n=\nU\n⋅\nΣ\n⋅\nV\nT\nWhat W = U·Σ·Vᵀ does: rotate the input (Vᵀ), stretch each direction by its singular value (Σ), rotate again (U).\nWhere each of the three matrices governs a distinct physical role in representation learning:\nRight Singular Vectors (\nV\nᵀ, the Input Frame):\nFunctions as the model’s input feature detector. It defines an orthonormal coordinate system that maps incoming token representations from context space into the model’s internal semantic subspaces.\nSingular Values (Σ, the Diagonal Spectrum):\nA non-negative diagonal matrix containing singular values σ₁ ≥ σ₂ ≥ … ≥ σᵣ ≥ 0. The spectrum dictates the\nenergy, scaling power, and intrinsic capacity\nallocated to each concept. This represents the model’s\nfundamental knowledge storage\n.\nLeft Singular Vectors (\nU\n, the Output Frame):\nThe output semantic projection. It maps features out of the latent space and routes them to the next transformer layer or the final vocabulary unembedding projection.\n2. What SFT Actually Does: Rewriting the Spectrum (ΔΣ >> 0)\nWhen you train a model with Supervised Fine-Tuning on next-token cross-entropy loss:\nL\nS\nF\nT\n(\nθ\n)\n=\n−\n∑\nt\n=\n1\nT\nlog\n⁡\nP\nθ\n(\ny\nt\n∣\ny\n<\nt\n,\nx\n)\n\\mathcal{L}_{\\mathrm{SFT}}(\\theta) = -\\sum_{t=1}^T \\log P_\\theta(y_t \\mid y_{<t}, x)\nL\nSFT\n​\n(\nθ\n)\n=\n−\nt\n=\n1\n∑\nT\n​\nlo\ng\nP\nθ\n​\n(\ny\nt\n​\n∣\ny\n<\nt\n​\n,\nx\n)\nThe objective forces the model to memorize target token distributions. If an enterprise wants a model to learn proprietary Verilog hardware libraries, internal legal compliance rules, or specialized medical terminology:\nNew Concepts Require New Energy:\nThe model must amplify representations along directions that were previously suppressed or dormant during pre-training.\nSpectrum Rescaling:\nSFT actively alters the diagonal matrix Σ. New singular values expand, existing ones shift, and the model’s spectral energy distribution is rewritten to store the injected domain knowledge.\nFrame Adaptation:\nConcurrently,\nU\nand\nV\nᵀ rotate to recognize new prompt formats and emit specific stylistic tokens.\nBecause SFT rewrites Σ, it is prone to\ncatastrophic forgetting\n: over-amplifying new singular values can collapse or distort pre-trained reasoning circuits.\n3. What RL Does: Spectral Inheritance (arXiv:2607.19331)\nIn reinforcement learning with verifiable rewards (such as GRPO applied to competitive math, coding, or formal verification), the loss is driven by outcome verification:\nL\nR\nL\nV\nR\n(\nθ\n)\n=\nE\n[\nπ\nθ\n(\na\n∣\ns\n)\nπ\no\nl\nd\n(\na\n∣\ns\n)\n⋅\nA\n(\ns\n,\na\n)\n]\n\\mathcal{L}_{\\mathrm{RLVR}}(\\theta) = \\mathbb{E}\\left[ \\frac{\\pi_\\theta(a \\mid s)}{\\pi_{\\mathrm{old}}(a \\mid s)} \\cdot A(s, a) \\right]\nL\nRLVR\n​\n(\nθ\n)\n=\nE\n[\nπ\nold\n​\n(\na\n∣\ns\n)\nπ\nθ\n​\n(\na\n∣\ns\n)\n​\n⋅\nA\n(\ns\n,\na\n)\n]\nIn their landmark paper\n“ISO: An RLVR-Native Optimization Stack”\n, Zhu et al. (UT Austin, Together AI) discovered a surprising empirical property:\nSpectral Inheritance\n.\nΣ\nR\nL\nV\nR\n≈\nΣ\n0\n\\Sigma_{\\mathrm{RLVR}} \\approx \\Sigma_0\nΣ\nRLVR\n​\n≈\nΣ\n0\n​\nWhy Does RL Preserve the Spectrum?\nReinforcement learning with outcome rewards cannot invent facts out of thin air. When an agent is rewarded for solving a multi-step geometry proof or finding a SQL join error, the reward signal does not provide external factual knowledge.\nInstead, RL solves a\nrouting and verification problem\n:\nThe underlying representation capacity (Σ₀) was already acquired during pre-training on trillions of tokens.\nRL optimizes the input singular frame\nV\nto better isolate critical problem constraints.\nRL rotates the output singular frame\nU\nto connect intermediate thinking steps to high-probability verification circuits (backtracking, error detection, self-correction).\nIn linear algebraic terms,\nreasoning is a rotation of coordinate frames, not an expansion of spectral energy\n.\n4. Isospectral Optimization (ISO): Training on Frames Alone\nZhu et al. turned this insight into a principled post-training paradigm called\nIsospectral Optimization (ISO)\n:\n1. ISO-Optimizer (Online RL):\nDuring training, freeze the base model’s singular values Σ₀ permanently. Run optimizers (like AdamW or Muon) exclusively over the frame variables (\nU\n,\nV\n):\nW\n(\nt\n)\n=\nU\n(\nt\n)\n⋅\nΣ\n0\n⋅\nV\n(\nt\n)\nT\nW(t) = U(t) \\cdot \\Sigma_0 \\cdot V(t)^{\\mathsf T}\nW\n(\nt\n)\n=\nU\n(\nt\n)\n⋅\nΣ\n0\n​\n⋅\nV\n(\nt\n)\nT\nOn Qwen3-8B, standard AdamW required 270 steps to reach 0.495 aggregate accuracy.\nISO-AdamW reached that same accuracy in only 100 steps (2.7x fewer steps)\nand continued improving to 0.509 at 210 steps, demonstrating that constraining optimization to the true manifold of reward-driven adaptation speeds up training.\n2. ISO-Merger (Offline Specialist Composition):\nIf two specialist models (e.g., a math agent and a coding agent) share the same base spectrum Σ₀, their learned capabilities live exclusively in their frame changes Δ\nU\nand Δ\nV\n. ISO-Merger composes these specialists directly in frame space without requiring rollouts, gradients, or distillation data.\nEmpirical Update (September 2026): Does ISO beat AdamW in practice?\nFollowing the ISO paper’s equations, we implemented ISO-AdamW and benchmarked it head-to-head against standard AdamW on a dedicated NVIDIA H200 GPU using GRPO on 1,000 held-out GSM8K math problems (Qwen3-1.7B-Base). While ISO-AdamW reached 75.8% accuracy versus AdamW’s 75.4% (+4 answers out of 1,000), it required +47.5% more peak VRAM and intensive polar projection iterations. Read our full end-to-end empirical report and systems post-mortem:\nWe Tried ISO-AdamW. AdamW Kept Its Job →\n5. Comparing SFT vs. RL Post-Training\nDimension\nSupervised Fine-Tuning (SFT)\nReinforcement Learning (RLVR / GRPO)\nPrimary Function\nKnowledge injection & style imitation\nReasoning, search exploration & verification\nWhat the Model Learns\n“What to say” (facts, syntax, vocabulary)\n“How to think” (search trees, backtracking, error recovery)\nWeight Spectrum (Σ)\nRewritten (ΔΣ >> 0)\n— allocates new singular values\nInherited (Σ ≈ Σ₀)\n— remains near-constant\nSingular Frames (\nU\n,\nV\n)\nAdapted to target surface style\nHeavily rotated\nto route logic into verifiers\nRisk Mode\nCatastrophic forgetting of pre-trained reasoning\nReward hacking or policy collapse if unconstrained\nData Requirement\nHigh-quality input-output demonstration pairs\nStateful environments with deterministic verifiers\n6. Practical Implications for Enterprise AI Systems\nUnderstanding this mathematical boundary changes how enterprise post-training pipelines should be built:\nNever Expect RL to Teach Unknown Facts:\nIf your base model has never seen your proprietary CAD kernel API, Modelica equations, or private database schemas, running GRPO will fail. Because RL cannot expand spectral capacity (Σ), it cannot discover facts out of vacuum.\nThe Governed Two-Stage Pipeline:\nStage 1 (SFT):\nUpload domain knowledge, grammar, and demonstration traces. This expands the spectrum Σ to encompass the domain vocabulary.\nStage 2 (RLVR in OpenEnv):\nConnect the adapted model to executable gyms with deterministic verifiers (lint, compiler, simulation). RL realigns the singular frames (\nU\n,\nV\n) to master reasoning, trade-off evaluation, and self-correction.\nApplying this to your own models?\nWe post-train open models on teams' own tasks with SFT and GRPO, and hand back the weights.\nLLM post-training with g factor\n.\nOriginally published at\ng-ftech.com\n.",
  "metadata": {
    "title": "SFT vs. RL: What Changes Inside the Model? - DEV Community",
    "description": "An intuitive guide to SFT, RLVR, and weight spectra: fine-tuning rewrites the singular values, RL keeps them and rotates the frames around them. Tagged with machinelearning, ai, llm, deeplearning.",
    "keywords": "machinelearning, ai, llm, deeplearning, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/g_factor/sft-vs-rl-what-changes-inside-the-model-30ho",
    "og:title": "SFT vs. RL: What Changes Inside the Model?",
    "og:description": "An intuitive guide to SFT, RLVR, and weight spectra: fine-tuning rewrites the singular values, RL keeps them and rotates the frames around them.",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fwww.g-ftech.com%2Fog%2Fsft-vs-rl-spectral-reasoning.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "SFT vs. RL: What Changes Inside the Model?",
    "twitter:description": "An intuitive guide to SFT, RLVR, and weight spectra: fine-tuning rewrites the singular values, RL keeps them and rotates the frames around them.",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fwww.g-ftech.com%2Fog%2Fsft-vs-rl-spectral-reasoning.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3aol0gthza4y73jqucwk.png",
      "bytes": 79098,
      "reused": true,
      "local_path": "attachments/45dff3e10a00/images/003.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3aol0gthza4y73jqucwk.png",
      "alt": "SFT grows the singular values Σ to store new facts. RLVR keeps Σ ≈ Σ₀ and rotates the frames U and Vᵀ instead.",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1lbu7ptgh3g82oua1zhy.png",
      "bytes": 53482,
      "reused": true,
      "local_path": "attachments/45dff3e10a00/images/004.png",
      "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1lbu7ptgh3g82oua1zhy.png",
      "alt": "What W = U·Σ·Vᵀ does: rotate the input (Vᵀ), stretch each direction by its singular value (Σ), rotate again (U).",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://arxiv.org/abs/2607.19331",
      "label": "arXiv:2607.19331, “ISO: An RLVR-Native Optimization Stack”",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3aol0gthza4y73jqucwk.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.g-ftech.com/blog/sft-vs-rl-spectral-reasoning?utm_source=devto&utm_medium=syndication",
      "label": "original post",
      "type": "a"
    },
    {
      "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1lbu7ptgh3g82oua1zhy.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.g-ftech.com/blog/iso-adamw-vs-adamw-grpo-experiment?utm_source=devto&utm_medium=syndication",
      "label": "We Tried ISO-AdamW. AdamW Kept Its Job →",
      "type": "a"
    },
    {
      "href": "https://www.g-ftech.com/solutions/llm-post-training?utm_source=devto&utm_medium=syndication",
      "label": "LLM post-training with g factor",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:57Z",
  "source_url": "https://dev.to/g_factor/sft-vs-rl-what-changes-inside-the-model-30ho",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2607.19331",
      "bytes": 8057093,
      "reused": false,
      "local_path": "papers/repair/2b75b3bbf7d6db6d96bb.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 9797,
  "expected_body_images": 2,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/45dff3e10a00/pages/source.html",
  "local_html": "attachments/45dff3e10a00/pages/article.html",
  "body_base_path": "attachments/45dff3e10a00/pages/",
  "body_markdown_path": "attachments/45dff3e10a00/pages/article.md"
}