{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Sebastian Raschka / Ahead of AI",
  "title_listed": "Understanding and Implementing Qwen3 From Scratch",
  "keywords": "Qwen3,开源模型,架构",
  "domain": "大模型算法",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://magazine.sebastianraschka.com/p/qwen3-from-scratch",
  "publish_date": "2026-08",
  "corpus_stage": "预训练/SFT",
  "notes": "Qwen3从零实现",
  "id": "c52b4cee69ff",
  "collected_at": "2026-10-01T20:32:59",
  "title": "Understanding and Implementing Qwen3 From Scratch",
  "body_markdown": "Previously, I compared the most notable open-weight architectures of 2025 in\nThe Big LLM Architecture Comparison\n. Then, I zoomed in and discussed the various architecture components in\nFrom GPT-2 to gpt-oss: Analyzing the Architectural Advances\non a conceptual level.\nSince all good things come in threes, before covering some of the noteworthy research highlights of this summer, I wanted to now dive into these architectures hands-on, in code. By following along, you will understand how it actually works under the hood and gain building blocks you can adapt for your own experiments or projects.\nFor this, I picked Qwen3 (\ninitially released in May\nand updated in July) because it is one of the most widely liked and used open-weight model families as of this writing.\nThe reasons why Qwen3 models are so popular are, in my view, as follows:\nA developer- and commercially friendly open-source (\nApache License v2.0\n) without any strings attached beyond the original open-source license terms (some other open-weight LLMs impose additional usage limits)\nThe performance is really good; for example, as of this writing, the open-weight 235B-Instruct variant is ranked 8 on the\nLMArena leaderboard\n, tied with the proprietary Claude Opus 4. The only 2 other open-weight LLMs that rank higher are DeepSeek 3.1 (3x larger) and Kimi K2 (4x larger). On September 5th,\nQwen3 released a 1T parameter “max” variant\non their platform that beats Kimi K2, DeepSeek 3.1, and Claude Opus 4 on all major benchmarks; however, this model is closed-source for now.\nThere are many different model sizes available for different compute budgets and use-cases, from 0.6B dense models to 480B parameter Mixture-of-Experts models.\nThis is going to be a long article due to the from-scratch code in pure PyTorch. While the code sections may look verbose, I hope that they help explain the building blocks better than conceptual figures alone!\nTip 1:\nIf you are reading this article in your email inbox, the narrow line width may cause code snippets to wrap awkwardly. For a better experience, I recommend\nopening it in your web browser\n.\nTip 2:\nYou can use the table of contents on the left side of the website for easier navigation between sections.\nFigure 1: Preview of the Qwen3 Dense and Mixture-of-Experts architectures discussed and (re)implemented in pure PyTorch in this article.",
  "metadata": {
    "title": "Understanding and Implementing Qwen3 From Scratch",
    "description": "A Detailed Look at One of the Leading Open-Source LLMs",
    "keywords": null,
    "author": "Sebastian Raschka, PhD",
    "og:type": "article",
    "og:title": "Understanding and Implementing Qwen3 From Scratch",
    "og:description": "A Detailed Look at One of the Leading Open-Source LLMs",
    "og:image": "https://substackcdn.com/image/fetch/$s_!0IOL!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00515b82-fd01-4c5b-b1b9-2d4b38100be0_1236x775.png",
    "og:url": "https://magazine.sebastianraschka.com/p/qwen3-from-scratch",
    "twitter:title": "Understanding and Implementing Qwen3 From Scratch",
    "twitter:description": "A Detailed Look at One of the Leading Open-Source LLMs",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!NLYJ!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F172832845%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2026-08-07T14:58:29.380Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": true,
    "marker": "this post is for paid subscribers",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!_APX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb3d5266-b65a-45ea-9f8f-98d7d8038b8e_3432x1536.png",
      "bytes": 130222,
      "reused": true,
      "local_path": "attachments/c52b4cee69ff/images/027.png",
      "src": "https://substackcdn.com/image/fetch/$s_!_APX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb3d5266-b65a-45ea-9f8f-98d7d8038b8e_3432x1536.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison",
      "label": "The Big LLM Architecture Comparison",
      "type": "a"
    },
    {
      "href": "https://magazine.sebastianraschka.com/p/from-gpt-2-to-gpt-oss-analyzing-the",
      "label": "From GPT-2 to gpt-oss: Analyzing the Architectural Advances",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2505.09388",
      "label": "initially released in May",
      "type": "a"
    },
    {
      "href": "https://huggingface.co/Qwen/Qwen3-0.6B/blob/main/LICENSE",
      "label": "Apache License v2.0",
      "type": "a"
    },
    {
      "href": "https://lmarena.ai/leaderboard/text",
      "label": "LMArena leaderboard",
      "type": "a"
    },
    {
      "href": "https://x.com/Alibaba_Qwen/status/1963991502440562976",
      "label": "Qwen3 released a 1T parameter “max” variant",
      "type": "a"
    },
    {
      "href": "https://magazine.sebastianraschka.com/p/qwen3-from-scratch",
      "label": "opening it in your web browser",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!_APX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb3d5266-b65a-45ea-9f8f-98d7d8038b8e_3432x1536.png",
      "label": "",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:26Z",
  "source_url": "https://magazine.sebastianraschka.com/p/qwen3-from-scratch",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2505.09388",
      "bytes": 779424,
      "reused": true,
      "local_path": "papers/repair/c08fa7046aa3f4744a1a.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 2366,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/c52b4cee69ff/pages/source.html",
  "local_html": "attachments/c52b4cee69ff/pages/article.html",
  "body_base_path": "attachments/c52b4cee69ff/pages/",
  "body_markdown_path": "attachments/c52b4cee69ff/pages/article.md"
}