{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Sebastian Raschka / Ahead of AI",
  "title_listed": "Accelerating PyTorch Model Training",
  "keywords": "PyTorch,训练加速,GPU",
  "domain": "机器学习",
  "content_type": "教程",
  "is_paid_flag": "否",
  "url": "https://magazine.sebastianraschka.com/p/accelerating-pytorch-model-training",
  "publish_date": "2022-09",
  "corpus_stage": "数据处理",
  "notes": "PyTorch训练加速技巧",
  "id": "467242a1d165",
  "collected_at": "2026-10-01T20:33:03",
  "title": "Accelerating PyTorch Model Training",
  "body_markdown": "Last week, I gave a talk on \"Scaling PyTorch Model Training With Minimal Code Changes\" at CVPR 2023 in Vancouver. It was really nice to chat with so many people at the event.\nFor future reference, and for those who couldn't attend, I wanted to try a little experiment and convert the talk into a short article, which you can find below! (PS: more on my takeaways from CVPR next week!)\nSo, this article delves into how to scale PyTorch model training with minimal code changes. The focus here is on leveraging mixed-precision techniques and multi-GPU training paradigms, not low-level machine optimizations. We will use a simple Vision Transformer (ViT) trained to classify images as our base model.\nStarting with a Baseline\nOur ViT, trained on a basic dataset from scratch, took about 60 minutes and achieved a test set accuracy of 62%.\nIn the upcoming sections, we will explore methods to improve the training time and accuracy without major code restructures.\nNote that the model and dataset details are not a primary concern here (they are just chosen to be as simple as possible so that you can replicate the code on your own hardware without downloading and installing too many dependencies.) All examples shared here are available in\nthis GitHub repository\n, where you can explore and reuse the complete code.\nOutput of the\n00_pytorch-vit-random-init.py\nscript.\nA Shift from Training from Scratch\nIn today's day an age, training deep learning models for text or vision from scratch is often wasteful. We usually leverage pretrained models and finetune them for our purposes to save time, computational resources, and obtain better modeling performance.\nIf we consider the same ViT architecture we used above, pretrained on a different dataset (ImageNet), and finetune it, we can achieve a far better prediction performance in much less time: 95% test accuracy in 20 min (3 training epochs).\nComparing\n00_pytorch-vit-random-init.py\nand\n01_pytorch-vit.py\nImproving Computational Performance\nAbove, we have seen that finetuning can give us a large boost over training from scratch, which is summarized in a more compact barplot below.\nComparing\n00_pytorch-vit-random-init.py\nand\n01_pytorch-vit.py\nOf course, our mileage may vary based on the dataset or task, but for many text and vision tasks, starting with a model pretrained on a general, public dataset is worthwhile.\nThe next sections will explore different tricks to accelerate the training time without sacrificing prediction accuracy.\nIntroducing the Open-Source Fabric Library\nOne way to efficiently scale training in PyTorch with minimal code changes is using the\nopen-source Fabric library\n, which can be considered a lightweight wrapper library/API around PyTorch. We install it via\npip install lightning\nBut of course, all the techniques we will explore below can also be implemented in pure PyTorch -- the goal of Fabric is to make this a bit more convenient.\nBefore we explore these \"advanced techniques to accelerate our code,\" let's introduce the small changes from incorporating Fabric into our PyTorch code. (Once we make these changes below, it will be easy to access advanced PyTorch features by changing only 1 line of code.)\nThe difference between simple PyTorch code and the modified one to use Fabric is subtle and involves only minor modifications, as highlighted in the code below:\nRegular PyTorch code (left) compared to PyTorch with Fabric (right)\nTo summarize the figure above, the main 3 steps for converting plain PyTorch code to PyTorch+Fabric are as follows:\nImport Fabric and instantiate a Fabric object.\nUse Fabric to set up the model, the optimizer, and the data loader.\nCall\nfabric.backward()\non the loss instead of the usual\nloss.backward()\nThese minor changes now provide a gateway to utilize advanced features in PyTorch without restructuring any more of the existing code.\nBut before we dive into these \"advanced features\" below, let's ensure that the model still has the same training runtime and predictive performance as before.\nComparison between\n01_pytorch-vit.py\nand\n03_fabric-vit.py\nAs we can see in the bar plots above, the training runtime and accuracies are exactly the same as before, as expected. (Any fluctuations can be attributed to randomness.)\nIn the previous section, we modified our PyTorch code using Fabric. Why go through all this hassle? As we will see below, we can now try advanced techniques, like mixed-precision and distributed training, by only changing one line of code, changing\nfabric = Fabric(accelerator=\"cuda\")\nto\nfabric = Fabric(accelerator=\"cuda\", precision=\"bf16-mixed\")\nResults from the\n04_fabric-vit-mixed-precision.py\nscript\nWith mixed precision training, we cut down the training time from about 18 minutes to 6 minutes while maintaining the same predictive performance. This reduction in training time was achieved simply by adding the argument\nprecision=\"bf16-mixed\"\nwhen instantiating the Fabric object.\nUnderstanding Mixed-Precision Mechanics\nMixed precision training essentially uses both 16-bit and 32-bit precision to ensure that there's no loss in accuracy. The computation of gradients in the 16-bit representation is much faster than in the 32-bit format and also saves a significant amount of memory. This strategy is beneficial, especially when we are memory or compute-constrained.\nIt's called \"mixed-\"rather than \"low-\"precision training because we don't transfer\nall\nparameters and operations to 16-bit floats. Instead, we switch between 32-bit and 16-bit operations during training, hence, the term \"mixed\" precision.\nAs illustrated in the figure below, mixed-precision training involves converting weights to lower-precision (FP16) for faster computation, calculating gradients, converting gradients back to higher-precision (FP32) for numerical stability, and updating the original weights with the scaled gradients.\nThis approach allows for efficient training while maintaining the accuracy and stability of the neural network.\nIn more detail, the steps are as follows.\nConvert weights to FP16: In this step, the weights (or parameters) of the neural network, which are initially in FP32 format, are converted to lower-precision FP16 format. This reduces the memory footprint and allows for faster computation, as FP16 operations require less memory and can be processed more quickly by the hardware.\nCompute gradients: The forward and backward passes of the neural network are performed using the lower-precision FP16 weights. This step calculates the gradients (partial derivatives) of the loss function with respect to the network’s weights, which are used to update the weights during the optimization process.\nConvert gradients to FP32: After computing the gradients in FP16, they are converted back to the higher-precision FP32 format. This conversion is essential for maintaining numerical stability and avoiding issues such as vanishing or exploding gradients that can occur when using lower-precision arithmetic.\nMultiply by learning rate and update weights: Now in FP32 format, the gradients are multiplied by a learning rate (a scalar value that determines the step size during optimization).\nThe product from step 4 is then used to update the original FP32 neural network weights. The learning rate helps control the convergence of the optimization process and is crucial for achieving good performance.\nBrain Float 16\nAbove, we talked about \"float 16-bit\" precision training. But notice that we specified\nprecision=\"bf16-mixed\"\ninstead of\nprecision=\"16-mixed\"\nin our code earlier -- both are valid options.\nHere, the \"bf16\" in\n\"bf16-mixed\"\nstands for\nBrain Floating Point\n(bfloat16). Google developed this format for machine learning and deep learning applications, particularly in their Tensor Processing Units (TPUs). Bfloat16 extends the dynamic range compared to the conventional float16 format at the expense of decreased precision.\nThe extended dynamic range helps bfloat16 to represent very large and very small numbers, making it more suitable for deep learning applications where a wide range of values might be encountered. However, the lower precision may affect the accuracy of certain calculations or lead to rounding errors in some cases. But in most deep learning applications, this reduced precision has minimal impact on modeling performance.\nWhile bfloat16 was originally developed for TPUs, this format is now supported by several NVIDIA GPUs as well, beginning with the A100 Tensor Core GPUs, which are part of the NVIDIA Ampere architecture.\nYou can check whether your GPU supports\nbfloat16\nvia the following code:\n>>> torch.cuda.is_bf16_supported()\nTrue\nIf your GPU does not support brain float 16, you can change\nprecision=\"bf16-mixed\"\nto\nprecision=\"16-mixed\"\n.\nMulti-GPU Training and Fully Sharded Data Parallelism\nThe next modification we are going to try is multi-GPU training. It becomes beneficial if we have multiple GPUs at our disposal since it allows us to train our models even faster.\nA more advanced technique that exploits this strategy is Fully Sharded Data Parallelism (FSDP), which utilizes both data parallelism and tensor parallelism.\nIn Fabric, we can leverage FSDP by adding the number of devices and multi-GPU training strategy as follows:\nfabric = Fabric(\n    accelerator=\"cuda\", precision=\"bf16-mixed\",\n    devices=4, strategy=\"FSDP\"  # new!\n)\nOutput of the\n06_fabric-vit-mixed-fsdp.py\nscript\nIt may be no big surprise since we are now involving 4 GPUs, but our code runs in about 2 minutes now, approximately 3 times faster than before when we only used mixed-precision training.\nUnderstanding Data Parallelism and Tensor Parallelism\nIn data parallelism, the mini-batch is divided, and a copy of the model is available on each of the GPUs. This process speeds up model training as multiple GPUs work in parallel.\nHere's how it works in a nutshell:\nThe same model is replicated across all the GPUs.\nEach GPU is then fed a different subset of the input data (a different mini-batch).\nAll GPUs independently perform forward and backward passes of the model, computing their own local gradients.\nThen, the gradients are collected and averaged across all GPUs.\nThe averaged gradients are then used to update the model's parameters.\nThe primary advantage of this approach is speed. Since each GPU is processing a unique mini-batch of data concurrently with the others, the model can be trained on more data in less time. This can significantly reduce the time required to train our model, especially when working with large datasets.\nHowever, data parallelism has some limitations. Most importantly, each GPU must have a complete copy of the model and its parameters. This places a limit on the size of the model we can train, as the model must fit within a single GPU's memory -- this is not feasible for modern ViTs or LLMs.\nUnlike data parallelism, which involves splitting a mini-batch across multiple devices, tensor parallelism divides the model itself across GPUs. In data parallelism, every GPU needs to fit the entire model, which can be a limitation when training larger models. Tensor parallelism, on the other hand, allows for training models that might be too large for a single GPU by breaking up the model and distributing it across multiple devices.\nHow does it work? Think of matrix multiplication. There are two ways to distribute it -- by row or by column. For simplicity, let's consider distribution by column. For instance, we can break down a large matrix multiplication operation into separate computations, each of which can be carried out on a different GPU, as shown in the figure below. The results are then concatenated to get the original result, effectively distributing the computational load.\nConclusion\nIn this article, we saw how we can tinker with techniques like mixed-precision training and distributed multi-GPU training by changing only a few lines of code. If we use mixed-precision training on the Vision Transformer example, we reduce the runtime 3 fold from ~18 min to ~6 min. And adding 3 more GPUs, we further reduced the training time from ~6 min to ~2 min. That's almost a 10-times improvement without sacrificing prediction accuracy.\nOf course, our mileage may vary based on the model, dataset, and task. However, the goal of this article was to mainly outline how we can experiment with these techniques conveniently using\nPyTorch\nand\nFabric\n.\n(For reference, all code examples are available\nhere on GitHub\n)\nThis magazine is a personal passion project that does not offer direct compensation. However, for those who wish to support me, please consider purchasing a copy of\none of my books\n. If you find them insightful and beneficial, please feel free to recommend them to your friends and colleagues.\nMachine Learning with PyTorch and Scikit-Learn\n,\nMachine Learning Q and AI\n, and\nBuild a Large Language Model (from Scratch)\nYour support means a great deal! Thank you!",
  "metadata": {
    "title": "Accelerating PyTorch Model Training",
    "description": "Maximize your PyTorch model's efficiency! Dive into mixed-precision & multi-GPU training techniques, with practical steps using the Fabric library. Learn how to achieve a 10x speed boost without losing prediction accuracy",
    "keywords": null,
    "author": "Sebastian Raschka, PhD",
    "og:type": "article",
    "og:title": "Accelerating PyTorch Model Training",
    "og:description": "Using Mixed-Precision and Fully Sharded Data Parallelism",
    "og:image": "https://substackcdn.com/image/fetch/$s_!4g8y!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9f9119e2-c9f6-4a13-a927-ffa66a80e485_862x844.png",
    "og:url": "https://magazine.sebastianraschka.com/p/accelerating-pytorch-model-training",
    "twitter:title": "Accelerating PyTorch Model Training",
    "twitter:description": "Using Mixed-Precision and Fully Sharded Data Parallelism",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!Ubrn!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F131034096%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2024-08-12T13:19:29.357Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/467242a1d165/code/body-000.txt",
      "chars": 21,
      "language": "text"
    },
    {
      "file": "attachments/467242a1d165/code/body-001.txt",
      "chars": 35,
      "language": "text"
    },
    {
      "file": "attachments/467242a1d165/code/body-002.txt",
      "chars": 59,
      "language": "text"
    },
    {
      "file": "attachments/467242a1d165/code/body-003.txt",
      "chars": 39,
      "language": "text"
    },
    {
      "file": "attachments/467242a1d165/code/body-004.txt",
      "chars": 105,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!MMuq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5548f31a-14a2-4ed3-96f3-d6ead0662d75_1458x1282.png",
      "bytes": 206953,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-f38e476c933485906968.png",
      "src": "https://substackcdn.com/image/fetch/$s_!MMuq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5548f31a-14a2-4ed3-96f3-d6ead0662d75_1458x1282.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!YlLI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92fab07e-a4ba-4218-8d52-a4e57e794c44_2064x914.png",
      "bytes": 182388,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-55f03aee9037a853fff5.png",
      "src": "https://substackcdn.com/image/fetch/$s_!YlLI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92fab07e-a4ba-4218-8d52-a4e57e794c44_2064x914.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!lrtS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe83295b5-274b-4ebc-9a1d-a9db18ed00b7_1958x724.png",
      "bytes": 65350,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-79ee295e9225429eb5a6.png",
      "src": "https://substackcdn.com/image/fetch/$s_!lrtS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe83295b5-274b-4ebc-9a1d-a9db18ed00b7_1958x724.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!0NiE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c3562e5-4ad5-487f-b9b8-8fc10634bf74_2204x1488.png",
      "bytes": 584193,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-1b7b2917ca349c3d569b.png",
      "src": "https://substackcdn.com/image/fetch/$s_!0NiE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c3562e5-4ad5-487f-b9b8-8fc10634bf74_2204x1488.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!9ihq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e41acb-7bc8-45ba-8a14-086256dbbe94_1882x1220.png",
      "bytes": 196302,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-51efa8dc164b32783bd8.png",
      "src": "https://substackcdn.com/image/fetch/$s_!9ihq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e41acb-7bc8-45ba-8a14-086256dbbe94_1882x1220.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!TfAN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93adbaeb-1d5b-4595-9db1-9e69ed6fe690_2162x624.png",
      "bytes": 51140,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-e0557297c28b10d13b3c.png",
      "src": "https://substackcdn.com/image/fetch/$s_!TfAN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93adbaeb-1d5b-4595-9db1-9e69ed6fe690_2162x624.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!cWxo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9ebc111e-319e-474f-a573-70306c61a53c_2286x928.png",
      "bytes": 84903,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-54ab85357bc0de343263.png",
      "src": "https://substackcdn.com/image/fetch/$s_!cWxo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9ebc111e-319e-474f-a573-70306c61a53c_2286x928.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!3v2p!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc81c2f9c-c950-4ff1-aace-3c3a0f61cd9c_1788x592.png",
      "bytes": 84992,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-9ee0f3e1cd6ede864056.png",
      "src": "https://substackcdn.com/image/fetch/$s_!3v2p!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc81c2f9c-c950-4ff1-aace-3c3a0f61cd9c_1788x592.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ehRJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0040301-df16-4306-9a50-af79b73f51ab_1640x1274.png",
      "bytes": 110959,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-8f74d5c7d0c08f3da722.png",
      "src": "https://substackcdn.com/image/fetch/$s_!ehRJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0040301-df16-4306-9a50-af79b73f51ab_1640x1274.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!cv6c!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe41fdec-d877-4e1b-83b9-65839cbaffc0_1842x1072.png",
      "bytes": 135745,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-753110ddd864546d4a87.png",
      "src": "https://substackcdn.com/image/fetch/$s_!cv6c!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe41fdec-d877-4e1b-83b9-65839cbaffc0_1842x1072.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!5ZkM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F495cce0a-6862-4e1a-8ff5-5879a568039c_2276x874.png",
      "bytes": 109404,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-589f28b67ca5af614c18.png",
      "src": "https://substackcdn.com/image/fetch/$s_!5ZkM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F495cce0a-6862-4e1a-8ff5-5879a568039c_2276x874.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!B3Nn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65192617-e486-406c-a64f-a4990457e387_2034x1016.png",
      "bytes": 80911,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-85c8a60c55aeb24693e0.png",
      "src": "https://substackcdn.com/image/fetch/$s_!B3Nn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65192617-e486-406c-a64f-a4990457e387_2034x1016.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!n6m1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f25e94-a68b-4adb-9241-01489d5c633b_1772x1002.png",
      "bytes": 89423,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-22df5839d70ae9050d91.png",
      "src": "https://substackcdn.com/image/fetch/$s_!n6m1!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f25e94-a68b-4adb-9241-01489d5c633b_1772x1002.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!d0kM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87d0967d-5661-4813-919d-de3531cdcda3_2154x1146.png",
      "bytes": 74415,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-4bd7c4846cf8db193e4a.png",
      "src": "https://substackcdn.com/image/fetch/$s_!d0kM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87d0967d-5661-4813-919d-de3531cdcda3_2154x1146.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!CxIZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27b6d27d-1052-4a4f-a419-a69cdd36704a_1118x454.png",
      "bytes": 77779,
      "reused": false,
      "local_path": "attachments/467242a1d165/images/body-4095da258ac44040a4a6.png",
      "src": "https://substackcdn.com/image/fetch/$s_!CxIZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27b6d27d-1052-4a4f-a419-a69cdd36704a_1118x454.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://github.com/rasbt/cvpr2023",
      "label": "this GitHub repository",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!MMuq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5548f31a-14a2-4ed3-96f3-d6ead0662d75_1458x1282.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/cvpr2023/blob/main/00_pytorch-vit-random-init.py",
      "label": "00_pytorch-vit-random-init.py",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!YlLI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92fab07e-a4ba-4218-8d52-a4e57e794c44_2064x914.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/cvpr2023/blob/main/01_pytorch-vit.py",
      "label": "01_pytorch-vit.py",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!lrtS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe83295b5-274b-4ebc-9a1d-a9db18ed00b7_1958x724.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://lightning.ai/docs/fabric/stable/",
      "label": "open-source Fabric library",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!0NiE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c3562e5-4ad5-487f-b9b8-8fc10634bf74_2204x1488.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!9ihq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27e41acb-7bc8-45ba-8a14-086256dbbe94_1882x1220.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!TfAN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93adbaeb-1d5b-4595-9db1-9e69ed6fe690_2162x624.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/cvpr2023/blob/main/03_fabric-vit.py",
      "label": "03_fabric-vit.py",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!cWxo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9ebc111e-319e-474f-a573-70306c61a53c_2286x928.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/cvpr2023/blob/main/04_fabric-vit-mixed-precision.py",
      "label": "04_fabric-vit-mixed-precision.py",
      "type": "a"
    },
    {
      "href": "https://cloud.google.com/tpu/docs/bfloat16",
      "label": "Brain Floating Point",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!ehRJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe0040301-df16-4306-9a50-af79b73f51ab_1640x1274.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!cv6c!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbe41fdec-d877-4e1b-83b9-65839cbaffc0_1842x1072.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!5ZkM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F495cce0a-6862-4e1a-8ff5-5879a568039c_2276x874.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/cvpr2023/blob/main/06_fabric-vit-mixed-fsdp.py",
      "label": "06_fabric-vit-mixed-fsdp.py",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!B3Nn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65192617-e486-406c-a64f-a4990457e387_2034x1016.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!n6m1!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3f25e94-a68b-4adb-9241-01489d5c633b_1772x1002.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!d0kM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F87d0967d-5661-4813-919d-de3531cdcda3_2154x1146.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://pytorch.org/",
      "label": "PyTorch",
      "type": "a"
    },
    {
      "href": "https://sebastianraschka.com/books",
      "label": "one of my books",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!CxIZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27b6d27d-1052-4a4f-a419-a69cdd36704a_1118x454.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Machine-Learning-PyTorch-Scikit-Learn-scikit-learn-ebook-dp-B09NW48MR1/dp/B09NW48MR1/",
      "label": "Machine Learning with PyTorch and Scikit-Learn",
      "type": "a"
    },
    {
      "href": "https://nostarch.com/machine-learning-and-ai-beyond-basics",
      "label": "Machine Learning Q and AI",
      "type": "a"
    },
    {
      "href": "http://mng.bz/M96o",
      "label": "Build a Large Language Model (from Scratch)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:47:26Z",
  "source_url": "https://magazine.sebastianraschka.com/p/accelerating-pytorch-model-training",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 12930,
  "expected_body_images": 15,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/467242a1d165/pages/source.html",
  "local_html": "attachments/467242a1d165/pages/article.html",
  "body_base_path": "attachments/467242a1d165/pages/",
  "body_markdown_path": "attachments/467242a1d165/pages/article.md"
}