{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Sebastian Raschka / Ahead of AI",
  "title_listed": "Coding LLMs from the Ground Up",
  "keywords": "代码LLM,Tokenizer,训练数据",
  "domain": "大模型算法",
  "content_type": "教程系列",
  "is_paid_flag": "否",
  "url": "https://magazine.sebastianraschka.com/p/coding-llms-from-the-ground-up",
  "publish_date": "2025-04",
  "corpus_stage": "预训练/SFT",
  "notes": "从零构建代码LLM，含数据清洗和训练流程",
  "id": "4b3449d49d4b",
  "collected_at": "2026-10-01T19:49:37",
  "title": "Coding LLMs from the Ground Up: A Complete Course",
  "body_markdown": "I wrote a lot about reasoning models in recent months (4 articles in a row)! Next to everything \"agentic,\" reasoning is one of the biggest LLM topics of 2025.\nThis month, however, I wanted to share more fundamental or \"foundational\" content with you on how to code LLMs, which is one of the best ways to understand how LLMs work.\nWhy? Many people really liked and benefited from the abbreviated LLM workshop I shared last year:\nBuilding LLMs from the Ground Up: A 3-hour Coding Workshop\nSebastian Raschka, PhD\n·\nAugust 31, 2024\nRead full story\nSo, I thought this ~5× longer and more detailed content (~15 hours in total) would be even more useful.\nAlso, I'm sadly dealing with a bad neck injury and haven't really been able to work on a computer for the past 3 weeks. I am currently trying a conservative treatment before considering the suggested surgical route. This is the worst timing as I just started to get back on track before life threw another curveball.\nSo, during my recovery, I thought sharing these videos I recorded in the last couple of months would be a nice in-between content.\nI hope you find this useful, and thanks for your support!\nPS: The videos originally started as supplementary content for my\nBuild a Large Language Model (From Scratch) book\n. But it turns out they also work pretty well as standalone content.\nWhy build from scratch?\nIt's probably the best and most efficient way to learn how LLMs really work. Plus, many readers have told me they had a lot of fun doing it.\nTo offer an analogy: if you are into cars and want to understand how they work, following a tutorial that walks you through building one from the ground up is a great way to learn. Of course, we probably wouldn't want to start by building a Formula 1 race car since it would be prohibitively expensive and overly complex for a first project. Instead, it makes more sense to start with something simpler, like a go-kart.\nBuilding a go-kart still teaches you how the steering works, how the motor functions, and more. You can even take it to the track and practice (and have a lot of fun with it) before stepping into a professional race car (or joining a company or team that is focused on building one). After all, the best race drivers often got their start by building and tinkering with their own go-karts (think Michael Schumacher and Ayrton Senna). By doing that, they not only developed a great feel for the car but could also provide valuable feedback to their mechanics, which gave them an edge over the other drivers.\nReferences\nBuild an LLM from Scratch book (\nManning\n|\nAmazon\n)\nBuild an LLM from Scratch GitHub repository\n1 - Set up your code environment (0:21:01)\nThis is a supplementary video explaining how to set up a Python environment using uv.\nIn particular, we are using “\nuv pip”\n, which is explained in\nthis document\n.\nAlternatively, the native “\nuv add”\nsyntax (mentioned but not explicitly covered in this video) is described\nhere\n.\nNote / Tip:\nThe installation may cause issues on certain versions of Windows. If you are on a Windows machine and have troubles with the installation (likely due to a TensorFlow dependency to load the original GPT-2 model weights from OpenAI in video 5), please don’t worry about it and feel free to skip the TensorFlow installation (you can do this by removing the TensorFlow line from the requirements file.)\nTo provide an alternative, I converted the GPT-2 model weights from a TensorFlow tensor format to PyTorch tensors and shared them on the Hugging Face model hub, which you can use as an alternative to the weight loading portion in video 5:\nhttps://huggingface.co/rasbt/gpt2-from-scratch-pytorch\n.\nIn any case, you don’t have to worry about this weight-loading code until the end of video 5.\n2 - Working with text data (1:28:01)\nThis video goes over text data preparations steps (tokenization, byte pair encoding, data loaders, etc.) for LLM training.\n3 - Coding attention mechanisms (2:15:40)\nThis is a supplementary video explaining how attention mechanisms (self-attention, causal attention, multi-head attention) work by coding them from scratch.\nYou can think of it as building the engine of a car (before adding the frame, seats, and wheels).\n4 - Set up your code environment (0:21:01)\nThis video covers how to code an LLM architecture from scratch.\n5 - Pretraining on Unlabeled Data (2:36:44)\nThis video explains how to pretrain a LLM from scratch.\n6 - Finetuning for Classification (2:15:29)\nThis is a video explaining how to fine-tune an LLM as a classifier (here using a spam classification example) as a gentle introduction to finetuning, before instruction finetuning the LLM in the next video.\n7 - Instruction Finetuning (1:46:04)\nFinally, this video explains how to instruction finetune the LLM.\nHappy viewing & tinkering!\nBonus: LLMs Then And Now (From 2018 to 2025)\nAs a big thank you to the paid subscribers, I want to share a 2.5h (non-coding) bonus video I recorded earlier in April, approximately 2 days after the Llama 4 release. In this talk, I discuss the current LLM landscape in 2025 with a focus on what and how things have changed since GPT-2 in 2018.\nThanks for your support, as an independent and self-employed researcher, this really means a lot to me!\nHopefully, things will improve in the next few weeks/months as I have lots of ideas for upcoming articles and can’t wait to work on them!",
  "metadata": {
    "title": "Coding LLMs from the Ground Up: A Complete Course",
    "description": "Why build an LLM from scratch? It's probably the best and most efficient way to learn how LLMs really work. Plus, many readers have told me they had a lot of fun doing it.",
    "keywords": null,
    "author": "Sebastian Raschka, PhD",
    "og:type": "article",
    "og:title": "Coding LLMs from the Ground Up: A Complete Course",
    "og:description": "Why build LLMs from scratch? It's probably the best and most efficient way to learn how LLMs really work. Plus, many readers have told me they had a lot of fun doing it.",
    "og:image": "https://substackcdn.com/image/fetch/$s_!DSfJ!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F752b6486-73db-4293-9465-854d4a005a5c_2406x1350.jpeg",
    "og:url": "https://magazine.sebastianraschka.com/p/coding-llms-from-the-ground-up",
    "twitter:title": "Coding LLMs from the Ground Up: A Complete Course",
    "twitter:description": "Why build LLMs from scratch? It's probably the best and most efficient way to learn how LLMs really work. Plus, many readers have told me they had a lot of fun doing it.",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!oHhY!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F162934205%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2025-06-16T21:46:27.528Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": true,
    "marker": "this post is for paid subscribers",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ye37!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F367b547c-9d22-4a3d-b466-1d56ccc6b055_1844x1224.png",
      "bytes": 5531,
      "reused": true,
      "local_path": "attachments/4b3449d49d4b/images/027.png",
      "src": "https://substackcdn.com/image/fetch/$s_!ye37!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F367b547c-9d22-4a3d-b466-1d56ccc6b055_1844x1224.png",
      "alt": "Building LLMs from the Ground Up: A 3-hour Coding Workshop",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=yAcWnfsZhzo",
      "src": "https://www.youtube-nocookie.com/embed/yAcWnfsZhzo?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/4b3449d49d4b/videos/repair-yAcWnfsZhzo.mp4",
      "bytes": 59593516,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "1261.221000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=341Rb8fJxY0",
      "src": "https://www.youtube-nocookie.com/embed/341Rb8fJxY0?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/4b3449d49d4b/videos/repair-341Rb8fJxY0.mp4",
      "bytes": 236972589,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "5280.601000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=-Ll8DtpNtvk",
      "src": "https://www.youtube-nocookie.com/embed/-Ll8DtpNtvk?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/4b3449d49d4b/videos/repair--Ll8DtpNtvk.mp4",
      "bytes": 358518182,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "8140.141000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=YSAkgEarBGE",
      "src": "https://www.youtube-nocookie.com/embed/YSAkgEarBGE?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/4b3449d49d4b/videos/repair-YSAkgEarBGE.mp4",
      "bytes": 298334300,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "6336.661000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=Zar2TJv-sE0",
      "src": "https://www.youtube-nocookie.com/embed/Zar2TJv-sE0?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/4b3449d49d4b/videos/repair-Zar2TJv-sE0.mp4",
      "bytes": 437868839,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "9403.061000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=5PFXJYme4ik",
      "src": "https://www.youtube-nocookie.com/embed/5PFXJYme4ik?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/4b3449d49d4b/videos/repair-5PFXJYme4ik.mp4",
      "bytes": 382100294,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "8128.101000",
      "reused": false
    },
    {
      "url": "https://www.youtube.com/watch?v=4yNswvhPWCQ",
      "src": "https://www.youtube-nocookie.com/embed/4yNswvhPWCQ?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/4b3449d49d4b/videos/repair-4yNswvhPWCQ.mp4",
      "bytes": 291314226,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "6363.161000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up",
      "label": "Building LLMs from the Ground Up: A 3-hour Coding Workshop Sebastian Raschka, PhD · August 31, 2024 Read full story",
      "type": "a"
    },
    {
      "href": "https://substack.com/profile/27393275-sebastian-raschka-phd",
      "label": "Sebastian Raschka, PhD",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4fqvn0D",
      "label": "Build a Large Language Model (From Scratch) book",
      "type": "a"
    },
    {
      "href": "https://mng.bz/M96o",
      "label": "Manning",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLMs-from-scratch",
      "label": "Build an LLM from Scratch GitHub repository",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLMs-from-scratch/blob/main/setup/01_optional-python-setup-preferences/README.md",
      "label": "this document",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLMs-from-scratch/blob/main/setup/01_optional-python-setup-preferences/native-uv.md",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://www.youtube-nocookie.com/embed/yAcWnfsZhzo?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://huggingface.co/rasbt/gpt2-from-scratch-pytorch",
      "label": "https://huggingface.co/rasbt/gpt2-from-scratch-pytorch",
      "type": "a"
    },
    {
      "href": "https://www.youtube-nocookie.com/embed/341Rb8fJxY0?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://www.youtube-nocookie.com/embed/-Ll8DtpNtvk?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://www.youtube-nocookie.com/embed/YSAkgEarBGE?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://www.youtube-nocookie.com/embed/Zar2TJv-sE0?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://www.youtube-nocookie.com/embed/5PFXJYme4ik?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://www.youtube-nocookie.com/embed/4yNswvhPWCQ?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "label": "",
      "type": "iframe"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:46:54Z",
  "source_url": "https://magazine.sebastianraschka.com/p/coding-llms-from-the-ground-up",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 5389,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/4b3449d49d4b/pages/source.html",
  "local_html": "attachments/4b3449d49d4b/pages/article.html",
  "body_base_path": "attachments/4b3449d49d4b/pages/",
  "body_markdown_path": "attachments/4b3449d49d4b/pages/article.md"
}