{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Sebastian Raschka / Ahead of AI",
  "title_listed": "Building LLMs from the Ground Up",
  "keywords": "LLM构建,Tokenizer,Transformer,训练",
  "domain": "大模型算法",
  "content_type": "教程系列",
  "is_paid_flag": "否",
  "url": "https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up",
  "publish_date": "2024-06",
  "corpus_stage": "预训练",
  "notes": "从零构建LLM的完整教程，信息密度极高",
  "id": "16d6a36a3560",
  "collected_at": "2026-10-01T19:49:13",
  "title": "Building LLMs from the Ground Up: A 3-hour Coding Workshop",
  "body_markdown": "If you’d like to spend a few hours this weekend to dive into Large Language Models (LLMs) and understand how they work, I've prepared a 3-hour coding workshop presentation on implementing, training, and using LLMs.\nBelow, you'll find a table of contents to get an idea of what this video covers (the video itself has clickable chapter marks, allowing you to jump directly to topics of interest):\n0:00 – Workshop overview\n2:17 – Part 1: Intro to LLMs\n9:14 – Workshop materials\n10:48 – Part 2: Understanding LLM input data\n23:25 – A simple tokenizer class\n41:03 – Part 3: Coding an LLM architecture\n45:01 – GPT-2 and Llama 2\n1:07:11 – Part 4: Pretraining\n1:29:37 – Part 5.1: Loading pretrained weights\n1:45:12 – Part 5.2: Pretrained weights via LitGPT\n1:53:09 – Part 6.1: Instruction finetuning\n2:08:21 – Part 6.2: Instruction finetuning via LitGPT\n02:26:45 – Part 6.3: Benchmark evaluation\n02:36:55 – Part 6.4: Evaluating conversational performance\n02:42:40 – Conclusion\nIt's a slight departure from my usual text-based content, but the last time I did this a few months ago, it was so well-received that I thought it might be nice to do another one!\nHappy viewing!\nReferences\nBuild an LLM from Scratch book\nBuild an LLM from Scratch GitHub repository\nGitHub repository with workshop code\nLightning Studio for this workshop\nLitGPT GitHub repository\nThis magazine is a personal passion project. For those who wish to support me, please consider purchasing a copy of my\nBuild a Large Language Model (From Scratch) book\n. (I am confident that you'll get lots out of this book as it explains how LLMs work in a level of detail that is not found anywhere else.)\nBuild a Large Language Model (From Scratch) now\navailable on Amazon\nIf you read the book and have a few minutes to spare, I'd really appreciate a\nbrief review\n. It helps us authors a lot!\nAlternatively, I also recently enabled the paid subscription option on Substack to support this magazine directly.\nAhead of AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.\nSubscribe",
  "metadata": {
    "title": "Building LLMs from the Ground Up: A 3-hour Coding Workshop",
    "description": "A 3-hour coding workshop presentation on implementing, training, and using LLMs.",
    "keywords": null,
    "author": "Sebastian Raschka, PhD",
    "og:type": "article",
    "og:title": "Building LLMs from the Ground Up: A 3-hour Coding Workshop",
    "og:description": "If your weekend plans include catching up on AI developments and understanding Large Language Models (LLMs), I've prepared a 1-hour presentation on the development cycle of LLMs, covering everything from architectural implementation to the finetuning stages.",
    "og:image": "https://substackcdn.com/image/fetch/$s_!ye37!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F367b547c-9d22-4a3d-b466-1d56ccc6b055_1844x1224.png",
    "og:url": "https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up",
    "twitter:title": "Building LLMs from the Ground Up: A 3-hour Coding Workshop",
    "twitter:description": "If your weekend plans include catching up on AI developments and understanding Large Language Models (LLMs), I've prepared a 1-hour presentation on the development cycle of LLMs, covering everything from architectural implementation to the finetuning stages.",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!WgxQ!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F148329414%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2024-12-21T23:07:47.778Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png",
      "bytes": 121470,
      "reused": false,
      "local_path": "attachments/16d6a36a3560/images/body-1c343ecc9eadd930f4bd.png",
      "src": "https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=quh7z1q7-uc",
      "src": "https://www.youtube-nocookie.com/embed/quh7z1q7-uc?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "status": "ok",
      "kind": "single_video",
      "local_path": "attachments/16d6a36a3560/videos/repair-quh7z1q7-uc.mp4",
      "bytes": 427055039,
      "valid": true,
      "height": 1080,
      "width": 1920,
      "duration": "9909.141000",
      "reused": false
    }
  ],
  "links": [
    {
      "href": "https://www.youtube-nocookie.com/embed/quh7z1q7-uc?rel=0&autoplay=0&showinfo=0&enablejsapi=0",
      "label": "",
      "type": "iframe"
    },
    {
      "href": "https://mng.bz/M96o",
      "label": "Build an LLM from Scratch book",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLMs-from-scratch",
      "label": "Build an LLM from Scratch GitHub repository",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/LLM-workshop-2024",
      "label": "GitHub repository with workshop code",
      "type": "a"
    },
    {
      "href": "https://lightning.ai/lightning-ai/studios/llms-from-the-ground-up-workshop",
      "label": "Lightning Studio for this workshop",
      "type": "a"
    },
    {
      "href": "https://github.com/Lightning-AI/litgpt",
      "label": "LitGPT GitHub repository",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4fqvn0D",
      "label": "Build a Large Language Model (From Scratch) book",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!woQp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167",
      "label": "brief review",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:32:37Z",
  "source_url": "https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 2102,
  "expected_body_images": 1,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/16d6a36a3560/pages/source.html",
  "local_html": "attachments/16d6a36a3560/pages/article.html"
}