[![Ahead of AI](https://substackcdn.com/image/fetch/$s_!96vs!,w_40,h_40,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49f25d0a-212b-4853-8bcb-128d0a3edbbf_1196x1196.png)](https://magazine.sebastianraschka.com/)
# [![Ahead of AI](https://substackcdn.com/image/fetch/$s_!xQ0c!,e_trim:10:white/e_trim:10:transparent/h_108,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5083e6d3-fbc9-4870-95b9-6e85d02f62a6_9366x2023.png)](https://magazine.sebastianraschka.com/)
SubscribeSign in
# Building LLMs from the Ground Up: A 3-hour Coding Workshop
[![Sebastian Raschka, PhD's avatar](https://substackcdn.com/image/fetch/$s_!CfW_!,w_36,h_36,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F61f4c017-506f-4e9b-a24f-76340dad0309_800x800.jpeg)](https://substack.com/@rasbt)
[Sebastian Raschka, PhD](https://substack.com/@rasbt)
Aug 31, 2024
446
16
28
Share
If you’d like to spend a few hours this weekend to dive into Large Language Models (LLMs) and understand how they work, I've prepared a 3-hour coding workshop presentation on implementing, training, and using LLMs.
Below, you'll find a table of contents to get an idea of what this video covers (the video itself has clickable chapter marks, allowing you to jump directly to topics of interest):
0:00 – Workshop overview
2:17 – Part 1: Intro to LLMs
9:14 – Workshop materials
10:48 – Part 2: Understanding LLM input data
23:25 – A simple tokenizer class
41:03 – Part 3: Coding an LLM architecture
45:01 – GPT-2 and Llama 2
1:07:11 – Part 4: Pretraining
1:29:37 – Part 5.1: Loading pretrained weights
1:45:12 – Part 5.2: Pretrained weights via LitGPT
1:53:09 – Part 6.1: Instruction finetuning
2:08:21 – Part 6.2: Instruction finetuning via LitGPT
02:26:45 – Part 6.3: Benchmark evaluation
02:36:55 – Part 6.4: Evaluating conversational performance
02:42:40 – Conclusion
It's a slight departure from my usual text-based content, but the last time I did this a few months ago, it was so well-received that I thought it might be nice to do another one!
**Happy viewing!**
## References
  1. [Build an LLM from Scratch book](https://mng.bz/M96o)
  2. [Build an LLM from Scratch GitHub repository](https://github.com/rasbt/LLMs-from-scratch)
  3. [GitHub repository with workshop code](https://github.com/rasbt/LLM-workshop-2024)
  4. [Lightning Studio for this workshop](https://lightning.ai/lightning-ai/studios/llms-from-the-ground-up-workshop)
  5. [LitGPT GitHub repository](https://github.com/Lightning-AI/litgpt)


* * *
_This magazine is a personal passion project. For those who wish to support me, please consider purchasing a copy of my[Build a Large Language Model (From Scratch) book](https://amzn.to/4fqvn0D). (I am confident that you'll get lots out of this book as it explains how LLMs work in a level of detail that is not found anywhere else.)_
[![](https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png)](https://substackcdn.com/image/fetch/$s_!woQp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png)Build a Large Language Model (From Scratch) now [available on Amazon](https://amzn.to/4fqvn0D)
_If you read the book and have a few minutes to spare, I'd really appreciate a[brief review](https://www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167). It helps us authors a lot!_
Alternatively, I also recently enabled the paid subscription option on Substack to support this magazine directly.
Ahead of AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Subscribe
[![Yann-Aël Le Borgne's avatar](https://substackcdn.com/image/fetch/$s_!Y7FF!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F28c8a381-0279-4b33-90ad-5f2f3405ae8b_96x96.png)](https://substack.com/profile/1137562-yann-ael-le-borgne)[![TAJ Haslani's avatar](https://substackcdn.com/image/fetch/$s_!03-Q!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fabd360e3-c362-4310-b037-4f5d3de178b6_144x144.png)](https://substack.com/profile/1565466-taj-haslani)[![Ihor's avatar](https://substackcdn.com/image/fetch/$s_!hax6!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6ea605a9-e5c3-461b-9f84-e8d7776720aa_144x144.png)](https://substack.com/profile/14042337-ihor)[![Jiri Fajtl, PhD's avatar](https://substackcdn.com/image/fetch/$s_!R7I8!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7089a088-52b4-4835-bd58-0877a500ca76_1024x1024.png)](https://substack.com/profile/6506035-jiri-fajtl-phd)[![Prayank Swaroop's avatar](https://substackcdn.com/image/fetch/$s_!p5U3!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f9aa15a-2fa2-4ca1-9022-50c6dc161e39_144x144.png)](https://substack.com/profile/314490-prayank-swaroop)
446 Likes∙
[28 Restacks](https://substack.com/note/p-148329414/restacks?utm_source=substack&utm_content=facepile-restacks)
446
16
28
Share
#### Discussion about this post
CommentsRestacks
![User's avatar](https://substackcdn.com/image/fetch/$s_!TnFC!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack.com%2Fimg%2Favatars%2Fdefault-light.png)
[![WvG's avatar](https://substackcdn.com/image/fetch/$s_!wIog!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F34b6f50e-88e9-46e2-b473-d1b60713ae0f_300x386.jpeg)](https://substack.com/profile/27977606-wvg?utm_source=comment)
[WvG](https://substack.com/profile/27977606-wvg?utm_source=substack-feed-item)
[Aug 31, 2024](https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up/comment/67351702 "Aug 31, 2024, 12:40 PM")
Liked by Sebastian Raschka, PhD
THANK YOU so much for doing this! I have bought the early release of your book via MEAP and it is fantastic. Highly recommended for everybody who wants to be hands on and really get a deeper understanding and appreciation regarding LLMs. 
[Like (8)](javascript:void\(0\))ReplyShare
[1 reply by Sebastian Raschka, PhD](https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up/comment/67351702)
[![Logan Thorneloe's avatar](https://substackcdn.com/image/fetch/$s_!wZCa!,w_32,h_32,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7d4c900b-c6e4-41d1-bf81-c7bb3794c6e3_500x500.png)](https://substack.com/profile/43759292-logan-thorneloe?utm_source=comment)
[Logan Thorneloe](https://substack.com/profile/43759292-logan-thorneloe?utm_source=substack-feed-item)
[Sep 4, 2024](https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up/comment/67804031 "Sep 4, 2024, 5:27 PM")
Liked by Sebastian Raschka, PhD
Thank you for the work you put into these! Do you mind if I add this to the machine learning road map I’ve put together as a guide to learn LLMs?
[Like (4)](javascript:void\(0\))ReplyShare
[1 reply by Sebastian Raschka, PhD](https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up/comment/67804031)
[14 more comments...](https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up/comments)
TopLatestDiscussions
[The Big LLM Architecture Comparison](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison)
[From DeepSeek V3 to GLM-5: A Look At Modern LLM Architecture Design](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison)
Jul 19, 2025 • [Sebastian Raschka, PhD](https://substack.com/@rasbt)
2,064
103
182
![](https://substackcdn.com/image/fetch/$s_!LmVE!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F45c50202-0e8b-4e64-8296-4e2ccf4cb287_1756x1227.png)
[Understanding Reasoning LLMs](https://magazine.sebastianraschka.com/p/understanding-reasoning-llms)
[Methods and Strategies for Building and Refining Reasoning Models](https://magazine.sebastianraschka.com/p/understanding-reasoning-llms)
Feb 5, 2025 • [Sebastian Raschka, PhD](https://substack.com/@rasbt)
1,368
47
127
![](https://substackcdn.com/image/fetch/$s_!QwUc!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd6ebc5c9-461f-4d3a-889b-b8ea4e14e5ba_1600x830.png)
[Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs](https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention)
[This article will teach you about self-attention mechanisms used in transformer architectures and large language models (LLMs) such as GPT-4 and Llama.](https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention)
Jan 14, 2024
528
43
20
![](https://substackcdn.com/image/fetch/$s_!3NS4!,w_320,h_213,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_center/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F69bfee26-ea3b-42a6-8a1a-6b8187852082_738x564.png)
See all
### Ready for more?
Subscribe
© 2026 Raschka AI Research (RAIR) Lab LLC · [Privacy](https://substack.com/privacy) ∙ [Terms](https://substack.com/tos) ∙ [Collection notice](https://substack.com/ccpa#personal-data-collected)
[ Start your Substack](https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer)[Get the app](https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button)
[Substack](https://substack.com) is the home for great culture