{
  "source_file": "0930",
  "source_platform": "Substack",
  "author": "Sebastian Raschka / Ahead of AI",
  "title_listed": "Improving LoRA and DoRA from Scratch",
  "keywords": "LoRA,DoRA,参数高效微调,PEFT",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://magazine.sebastianraschka.com/p/lora-and-dora-from-scratch",
  "publish_date": "2024-05",
  "corpus_stage": "SFT",
  "notes": "从原理推导LoRA/DoRA，含代码实现和实验对比",
  "id": "9d0f12f4323c",
  "collected_at": "2026-10-01T19:49:31",
  "title": "Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch",
  "body_markdown": "Low-rank adaptation (LoRA) is a machine learning technique that modifies a pretrained model (for example, an LLM or vision transformer) to better suit a specific, often smaller, dataset by adjusting only a small, low-rank subset of the model's parameters.\nThis approach is important because it allows for efficient finetuning of large models on task-specific data, significantly reducing the computational cost and time required for finetuning.\nLast week, researchers proposed\nDoRA: Weight-Decomposed Low-Rank Adaptation\n, a new alternative to LoRA, which may outperform LoRA by a large margin.\nDoRA is a promising alternative to standard LoRA (annotated figure from the DoRA paper:\nhttps://arxiv.org/abs/2402.09353\n)\nTo understand how these methods work, we will implement both LoRA and DoRA in PyTorch from scratch in this article!\nLoRA Recap\nBefore we dive into DoRA, here's a brief recap of how\nLoRA\nworks.\nSince LLMs are large, updating all model weights during training can be expensive due to GPU memory limitations. Suppose we have a large weight matrix\nW\nfor a given layer. During backpropagation, we learn a\nΔW\nmatrix, which contains information on how much we want to update the original weights to minimize the loss function during training.\nIn regular training and finetuning, the weight update is defined as follows:\nW\nupdated\n= W + ΔW\nThe LoRA method proposed by\nHu\net al.\noffers a more efficient alternative to computing the weight updates\nΔW\nby learning an approximation of it,\nΔW ≈ AB\n. In other words, in LoRA, we have the following, where\nA\nand\nB\nare two small weight matrices:\nW\nupdated\n= W + A.B\n(The \"\n.\n\" in \"\nA.B\n\" stands for matrix multiplication.)\nThe figure below illustrates these formulas for full finetuning and LoRA side by side.\nFigure: An illustration of regular finetuning (left) and LoRA finetuning (right).\nHow does LoRA save GPU memory? If a pretrained weight matrix\nW\nis a 1,000×1,000 matrix, then the weight update matrix\nΔW\nin regular finetuning is a 1,000×1,000 matrix as well. In this case,\nΔW\nhas 1,000,000 parameters. If we consider a LoRA rank of 2, then\nA\nis a 1000×2 matrix, and\nB\nis a 2×1000 matrix, and we only have 2×2×1,000 = 4,000 parameters that we need to update when using LoRA. In the previous example, with a rank of 2, that's 250 times fewer parameters.\nOf course,\nA\nand\nB\ncan't capture all the information that\nΔW\ncould capture, but this is by design. When using LoRA, we hypothesize that the model requires\nW\nto be a large matrix with full rank to capture all the knowledge in the pretraining dataset. However, when we finetune an LLM, we don't need to update all the weights and capture the core information for the adaptation in a smaller number of weights than\nΔW\nwould; hence, we have the low-rank updates via\nAB\n.\nIf you paid close attention, the full finetuning and LoRA depictions in the figure above look slightly different from the formulas I have shown earlier. That's due to the distributive law of matrix multiplication: we don't have to add the weights with the updated weights but can keep them separate. For instance, if\nx\nis the input data, then we can write the following for regular finetuning:\nx.(W+ΔW) = x.W + x.ΔW\nSimilarly, we can write the following for LoRA:\nx.(W+A.B)\n=\nx.W + x.A.B\nThe fact that we can keep the LoRA weight matrices separate makes LoRA especially attractive. In practice, this means that we don't have to modify the weights of the pretrained model at all, as we can apply the LoRA matrices on the fly. This is especially useful if you are considering hosting a model for multiple customers. Instead of having to save the large updated models for each customer, you only have to save a small set of LoRA weights alongside the original pretrained model.\nTo make this less abstract and to provide additional intuition, we will implement LoRA in code from scratch in the next section.\nA LoRA Layer Code Implementation\nWe begin by initializing a\nLoRALayer\nthat creates the matrices A and B, along with the alpha scaling hyperparameter and the rank hyperparameters. This layer can accept an input and compute the corresponding output, as illustrated in the figure below.\nIllustration of the LoRA matrices A and B with rank\nr\n.\nIn code, this LoRA layer depicted in the figure above looks like as follows:\nimport torch.nn as nn\n\nclass LoRALayer(nn.Module):\n    def __init__(self, in_dim, out_dim, rank, alpha):\n        super().__init__()\n        std_dev = 1 / torch.sqrt(torch.tensor(rank).float())\n        self.A = nn.Parameter(torch.randn(in_dim, rank) * std_dev)\n        self.B = nn.Parameter(torch.zeros(rank, out_dim))\n        self.alpha = alpha\n\n    def forward(self, x):\n        x = self.alpha * (x @ self.A @ self.B)\n        return x\nIn the code above,\nrank\nis a hyperparameter that controls the inner dimension of the matrices\nA\nand\nB\n. In other words, this parameter controls the number of additional parameters introduced by LoRA and is a key factor in determining the balance between model adaptability and parameter efficiency.\nThe second hyperparameter,\nalpha\n, is a scaling hyperparameter applied to the output of the low-rank adaptation. It essentially controls the extent to which the adapted layer's output is allowed to influence the original output of the layer being adapted. This can be seen as a way to regulate the impact of the low-rank adaptation on the layer's output.\nSo far, the\nLoRALayer\nclass we implemented above allows us to transform the layer inputs\nx\n. However, in LoRA, we are usually interested in replacing existing\nLinea\nr layers so that the weight update is applied to the existing pretrained weights, as shown in the figure below:\nLoRA applied to an existing linear layer\nTo incorporate the original Linear layer weights as shown in the figure above, we will implement a\nLinearWithLoRA\nlayer that uses the previously implemented\nLoRALayer\nand can be used to replace existing\nLinea\nr layers in a neural network, for example, the self-attention module or feed forward modules in an LLM:\nclass LinearWithLoRA(nn.Module):\n\n    def __init__(self, linear, rank, alpha):\n        super().__init__()\n        self.linear = linear\n        self.lora = LoRALayer(\n            linear.in_features, linear.out_features, rank, alpha\n        )\n\n    def forward(self, x):\n        return self.linear(x) + self.lora(x)\nNote that since we initialize the weight matrix B (\nself.B\nin\nLoraLayer\n) with zero values in the LoRA layer, the matrix multiplication between\nA\nand\nB\nresults in a matrix consisting of 0's and doesn't affect the original weights (since adding 0 to the original weights does not modify them).\nLet's try out LoRA on a small neural network layer represented by a single\nLinear\nlayer:\nIn:\ntorch.manual_seed(123)\nlayer = nn.Linear(10, 2)\nx = torch.randn((1, 10))\n\nprint(\"Original output:\", layer(x))\nOut:\nOriginal output: tensor([[0.6639, 0.4487]], grad_fn=<AddmmBackward0>)\nNow, applying LoRA to the\nLinea\nr layer, we see that the results are the same since we haven't trained the LoRA weights yet. In other words, everything works as expected:\nIn:\nlayer_lora_1 = LinearWithLoRA(layer, rank=2, alpha=4)\nprint(\"LoRA output:\", layer_lora_1(x))\nOut:\nLoRA output: tensor([[0.6639, 0.4487]], grad_fn=<AddmmBackward0>)\nEarlier, I mentioned the distributive law of matrix multiplication:\nx.(W+A.B)\n=\nx.W + x.A.B\n.\nHere, this means that we can also combine or merge the LoRA matrices and original weights, which should result in an equivalent implementation. In code, this alternative implementation to the\nLinearWithLoRA\nlayer looks as follows:\nclass LinearWithLoRAMerged(nn.Module):\n    def __init__(self, linear, rank, alpha):\n        super().__init__()\n        self.linear = linear\n        self.lora = LoRALayer(\n            linear.in_features, linear.out_features, rank, alpha\n        )\n\n    def forward(self, x):\n        lora = self.lora.A @ self.lora.B # Combine LoRA matrices\n        # Then combine LoRA with orig. weights\n        combined_weight = self.linear.weight + self.lora.alpha*lora.T \n        return F.linear(x, combined_weight, self.linear.bias)\nIn short,\nLinearWithLoRAMerged\ncomputes the left side of the equation\nx.(W+A.B)\n=\nx.W + x.A.B\nwhereas\nLinearWithLoRA\ncomputes the right side -- both are equivalent.\nWe can verify that this results in the same outputs as before via the following code:\nIn:\nlayer_lora_2 = LinearWithLoRAMerged(layer, rank=2, alpha=4)\nprint(\"LoRA output:\", layer_lora_2(x))\nOut:\nLoRA output: tensor([[0.6639, 0.4487]], grad_fn=<AddmmBackward0>)\nNow that we have a working LoRA implementation let's see how we can apply it to a neural network in the next section.\nAhead of AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.\nSubscribe\nApplying LoRA Layers\nWhy did we implement LoRA in the manner described above using PyTorch modules? This approach enables us to easily replace a\nLinear\nlayer in an existing neural network (for example, the feed forward or attention modules of a Large Language Model) with our new\nLinearWithLoRA\n(or\nLinearWithLoRAMerged\n) layers.\nFor simplicity, let's focus on a small 3-layer multilayer perceptron instead of an LLM for now, which is illustrated in the figure below:\nA simple 3-layer multilayer perceptron\nIn code, we can implement the multilayer perceptron, shown above, as follows:\nIn:\nclass MultilayerPerceptron(nn.Module):\n    def __init__(self, num_features, \n        num_hidden_1, num_hidden_2, num_classes):\n        super().__init__()\n        self.layers = nn.Sequential(\n            nn.Linear(num_features, num_hidden_1),\n            nn.ReLU(),\n            nn.Linear(num_hidden_1, num_hidden_2),\n            nn.ReLU(),\n\n            nn.Linear(num_hidden_2, num_classes)\n        )\n\n    def forward(self, x):\n        x = self.layers(x)\n        return x\n\n\nmodel = MultilayerPerceptron(\n    num_features=num_features,\n    num_hidden_1=num_hidden_1,\n    num_hidden_2=num_hidden_2, \n    num_classes=num_classes\n)\n\nprint(model)\nOut:\nMultilayerPerceptron(\n  (layers): Sequential(\n    (0): Linear(in_features=784, out_features=128, bias=True)\n    (1): ReLU()\n    (2): Linear(in_features=128, out_features=256, bias=True)\n    (3): ReLU()\n    (4): Linear(in_features=256, out_features=10, bias=True)\n  )\n)\nUsing\nLinearWithLora\n, we can then add the LoRA layers by replacing the original\nLinear\nlayers in the multilayer perceptron model:\nIn:\nmodel.layers[0] = LinearWithLoRA(model.layers[0], rank=4, alpha=8)\nmodel.layers[2] = LinearWithLoRA(model.layers[2], rank=4, alpha=8)\nmodel.layers[4] = LinearWithLoRA(model.layers[4], rank=4, alpha=8)\n\nprint(model)\nOut:\nMultilayerPerceptron(\n  (layers): Sequential(\n    (0): LinearWithLoRA(\n      (linear): Linear(in_features=784, out_features=128, bias=True)\n      (lora): LoRALayer()\n    )\n    (1): ReLU()\n    (2): LinearWithLoRA(\n      (linear): Linear(in_features=128, out_features=256, bias=True)\n      (lora): LoRALayer()\n    )\n    (3): ReLU()\n    (4): LinearWithLoRA(\n      (linear): Linear(in_features=256, out_features=10, bias=True)\n      (lora): LoRALayer()\n    )\n  )\n)\nThen, we can freeze the original\nLinear\nlayers and only make the\nLoRALaye\nr layers trainable, as follows:\nIn:\ndef freeze_linear_layers(model):\n    for child in model.children():\n        if isinstance(child, nn.Linear):\n            for param in child.parameters():\n                param.requires_grad = False\n        else:\n            # Recursively freeze linear layers in children modules\n            freeze_linear_layers(child)\n\nfreeze_linear_layers(model)\nfor name, param in model.named_parameters():\n    print(f\"{name}: {param.requires_grad}\")\nOut:\nlayers.0.linear.weight: False\nlayers.0.linear.bias: False\nlayers.0.lora.A: True\nlayers.0.lora.B: True\nlayers.2.linear.weight: False\nlayers.2.linear.bias: False\nlayers.2.lora.A: True\nlayers.2.lora.B: True\nlayers.4.linear.weight: False\nlayers.4.linear.bias: False\nlayers.4.lora.A: True\nlayers.4.lora.B: True\nBased on the\nTrue\nand\nFalse\nvalues above, we can visually confirm that only the LoRA layers are trainable now (\nTrue\nmeans trainable,\nFalse\nmeans frozen). In practice, we would then train the network with this LoRA configuration on a new dataset or task.\nTo avoid making this a very long article, I am skipping over the boilerplate code to train this model. But if you are interested in the full code, you can find a standalone code notebook here:\nhttps://github.com/rasbt/dora-from-scratch\n.\nFurthermore, if you are interested in a LoRA from scratch explanation and application to an LLM, also check out my Lightning Studio\nLoRA From Scratch – Implement Low-Rank Adaptation for LLMs in PyTorch\n.\nUnderstanding Weight-Decomposed Low-Rank Adaptation (DoRA)\nYou may have noticed that we spent a lot of time implementing and talking about LoRA. That's because DoRA (\nWeight-Decomposed Low-Rank Adaptation\n) can be seen as an improvement or extension of LoRA that is built on top of it, and we can now easily adapt some of our previous code to implement DoRA.\nDoRA can be described in two steps, where the first step is to decompose a pretrained weight matrix into a magnitude vector (\nm\n) and a directional matrix (\nV\n). The second step is applying LoRA to the directional matrix\nV\nand training the magnitude vector\nm\nseparately.\nThe decomposition into  magnitude and directional components is inspired by the mathematical principle that any vector can be represented as the product of its magnitude (a scalar value indicating its length) and its direction (a unit vector indicating its orientation in space).\nIllustration of the direction and magnitude of a single vector. For example, if have a 2D vector [1, 2], we can decompose it into a magnitude 2.24 and a directional vector [0.447,  0.894]. Then 2.24 * [0.447, 0.894] = [1, 2].\nIn DoRA, we apply the decomposition into magnitude and directional components to a whole pretrained weight matrix\nW\ninstead of a vector, where each column (vector) of the weight matrix corresponds to the weights connecting all inputs to a particular output neuron.\nSo, the result of decomposing\nW\nis a magnitude vector\nm\nthat represents the scale or length of each column vector in the weight matrix, as illustrated in the figure below.\nIllustration of the weight matrix decomposition in DoRA\nThen, DoRA takes the directional matrix\nV\nand applies standard LoRA, for instance:\nW' = m (V + ΔV)/norm = m (W + AB)/norm\nThe normalization, which I abbreviated as \"norm\" to not further complicate things in this overview, is based on the weight normalization method proposed in Saliman's and Kingma's 2016\nWeight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks paper\n.\nThe DoRA two-step process (decomposing a pretrained weight matrix and applying LoRA to the directional matrix) is further illustrated in the figure from the DoRA paper below.\nAnnotated illustration from the DoRA paper (\nhttps://arxiv.org/abs/2402.09353\n)\nThe motivation for developing DoRA is based on analyzing and comparing the LoRA and full finetuning learning patterns. The DoRA authors found that LoRA either increases or decreases magnitude and direction updates proportionally but seems to lack the capability to make only subtle directional changes as found in full finetuning. Hence, the researchers propose the decoupling of magnitude and directional components.\nIn other words, their DoRA method aims to apply LoRA only to the directional component,\nV\n, while also allowing the magnitude component,\nm\n, to be trained separately.\nIntroducing the magnitude vector m adds 0.01% more parameters if DoRA is compared to LoRA. However, across both LLM and vision transformer benchmarks, they found that DoRA even outperforms LoRA if the DoRA rank is halved, for instance, when DoRA only uses half the parameters of regular LoRA, as shown in the performance comparison below.\nComparison between LoRA and DoRA from the DoRA paper (\nhttps://arxiv.org/abs/2402.09353\n)\nAs I wrote in another article a few months ago, LoRA requires careful tuning of the rank to optimize performance:\nPractical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation\n). However, DoRA seems to be much more robust to changes in rank, as shown in the comparison below.\nDoRA is more robust to the rank hyperparameter than LoRA (annotated figure from the DoRA paper,\nhttps://arxiv.org/abs/2402.09353\n)\nThe possibility to successfully use DoRA with relatively small ranks makes this method even more parameter-efficient than LoRA.\nOverall, I am quite impressed by the results, and it should not be too big of a lift to upgrade a LoRA implementation to DoRA, which we will do in the next section.\nImplementing DoRA Layers in PyTorch\nIn this section, we will see what DoRA looks like in code. Previously, we said that we can initialize a pretrained weight\nW\n0\nwith magnitude\nm\nand directional component\nV\n. For instance, we have the following equation:\n\\(W_0 = m \\frac{V}{||V||_c} \\)\nwhere ||\nV\n||\nc\nis the vector-wise norm of\nV\n. Then we can write DoRA including the LoRA weight update\nBA\nas shown below:\n\\(W^{\\prime}=m \\frac{V+BA}{\\|V+BA\\|_c}\\)\nNow, in the DoRA paper, the authors formulate DoRA as follows, where they use the initial pretrained weights\nW\n0\nas the directional component directly and learn magnitude vector\nm\nduring training:\n\\(W^{\\prime}={m} \\frac{V+\\Delta V}{\\|V+\\Delta V\\|_c}={m} \\frac{W_0+{B A}}{\\left\\|W_0+{B A}\\right\\|_c}\\)\nHere,\nΔV\nis the update to the directional component, matrix\nV\n.\nWhile the original authors haven't released the official implementation yet, you can find an independent implementation\nhere\n, which loosely inspired my implementation below\nTaking our previous\nLinearWithLoRAMerged\nimplementation, we can upgrade it to DoRA as follows:\nclass LinearWithDoRAMerged(nn.Module):\n\n    def __init__(self, linear, rank, alpha):\n        super().__init__()\n        self.linear = linear\n        self.lora = LoRALayer(\n            linear.in_features, linear.out_features, rank, alpha\n        )\n        self.m = nn.Parameter(\n            self.linear.weight.norm(p=2, dim=0, keepdim=True))\n\n  \n  # Code loosely inspired by    \n  # https://github.com/catid/dora/blob/main/dora.py\n\n    def forward(self, x):\n        lora = self.lora.A @ self.lora.B\n        numerator = self.linear.weight + self.lora.alpha*lora.T\n        denominator = numerator.norm(p=2, dim=0, keepdim=True)\n        directional_component = numerator / denominator\n        new_weight = self.m * directional_component\n        return F.linear(x, new_weight, self.linear.bias)\nThe\nLinearWithDoRAMerged\nclass is different from our previous\nLinearWithLoRAMerged\nclass in several key aspects, primarily in how it modifies and applies the weights of the Linear layer. However, both classes integrate a\nLoRALayer\nto augment the original linear layer's weights, but DoRA adds weight normalization and adjustment.\nThe figure below shows a file-diff of both classes side by side:\nFile-diff between\nLinearWithLoRAMerged\nand\nLinearWithDoRAMerged\nAs we can see in the figure above,\nLinearWithDoRAMerged\nintroduces an additional step involving dynamic normalization of the augmented weights.\nAfter combining the original weights with the LoRA-adjusted weights (\nself.linear.weight + self.lora.alpha*lora.T\n), it calculates the norm of these combined weights across columns (\ncolumn_norm\n). Then, it normalizes the combined weights by dividing them by their norms (\nV = combined_weight / column_norm\n). This step ensures that each column of the combined weight matrix has a unit norm, which can help stabilize the learning process by maintaining the scale of weight updates.\nDoRA also introduces a learnable vector\nself.m\n, which represents the magnitude of each column of the normalized weight matrix. This parameter allows the model to dynamically adjust the scale of each weight vector in the combined weight matrix during training. This additional flexibility can help the model better capture the importance of different features.\nIn summary,\nLinearWithDoRAMerged\nextends the concept of\nLinearWithLoRAMerged\nby incorporating dynamic weight normalization and scaling to improve the training performance.\nIn practice, considering the multilayer perceptron from earlier, we can simply swap existing Linear layers with our\nLinearWithDoRAMerged\nlayers as follows:\nIn:\nmodel.layers[0] = LinearWithDoRAMerged(model.layers[0], rank=4, alpha=8)\nmodel.layers[2] = LinearWithDoRAMerged(model.layers[2], rank=4, alpha=8)\nmodel.layers[4] = LinearWithDoRAMerged(model.layers[4], rank=4, alpha=8)\n\nprint(model)\nOut:\nMultilayerPerceptron(\n  (layers): Sequential(\n    (0): LinearWithDoRAMerged(\n      (linear): Linear(in_features=784, out_features=128, bias=True)\n      (lora): LoRALayer()\n    )\n    (1): ReLU()\n    (2): LinearWithDoRAMerged(\n      (linear): Linear(in_features=128, out_features=256, bias=True)\n      (lora): LoRALayer()\n    )\n    (3): ReLU()\n    (4): LinearWithDoRAMerged(\n      (linear): Linear(in_features=256, out_features=10, bias=True)\n      (lora): LoRALayer()\n    )\n  )\n)\nBefore we finetune the model, we can reuse the\nfreeze_linear_layers\nfunction we implemented earlier to only make the LoRA weights and magnitude vectors trainable:\nIn:\nfreeze_linear_layers(model)\nfor name, param in model.named_parameters():\n    print(f\"{name}: {param.requires_grad}\")\nOut:\nlayers.0.m: True\nlayers.0.linear.weight: False\nlayers.0.linear.bias: False\nlayers.0.lora.A: True\nlayers.0.lora.B: True\nlayers.2.m: True\nlayers.2.linear.weight: False\nlayers.2.linear.bias: False\nlayers.2.lora.A: True\nlayers.2.lora.B: True\nlayers.4.m: True\nlayers.4.linear.weight: False\nlayers.4.linear.bias: False\nlayers.4.lora.A: True\nlayers.4.lora.B: True\nThe full code example, including model training, is available in my GitHub repo here:\nhttps://github.com/rasbt/dora-from-scratch\n.\nDoRA code notebook:\nhttps://github.com/rasbt/dora-from-scratch\nConclusion\nIn my opinion, DoRA seems like a logical, effective, and promising extension of LoRA, and I am excited to try it in real-world LLM finetuning contexts.\nIn the meantime, I also added the DoRA implementation above to the\nLoRA From Scratch – Implement Low-Rank Adaptation for LLMs in PyTorch\nLightning Studio to finetune a DistilBERT language model (see\nbonus_02_finetune-with-dora.ipynb\n). Even without hyperparameter tuning, I already saw a >1% prediction accuracy improvement over LoRA.\nThis magazine is a personal passion project. For those who wish to support me, please consider purchasing a copy of my\nBuild a Large Language Model (From Scratch) book\n. (I am confident that you'll get lots out of this book as it explains how LLMs work in a level of detail that is not found anywhere else.)\nBuild a Large Language Model (From Scratch) now\navailable on Amazon\nIf you read the book and have a few minutes to spare, I'd really appreciate a\nbrief review\n. It helps us authors a lot!\nAlternatively, I also recently enabled the paid subscription option on Substack to support this magazine directly.\nAhead of AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.\nSubscribe",
  "metadata": {
    "title": "Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch",
    "description": "This article implements LoRA (low-rank adaptation), an parameter-efficient finetuning technique for LLMs from scratch and discussed the newest and most promising variant: DoRA (Weight-Decomposed Low-Rank Adaptation).",
    "keywords": null,
    "author": "Sebastian Raschka, PhD",
    "og:type": "article",
    "og:title": "Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch",
    "og:description": "Low-rank adaptation (LoRA) is a machine learning technique that modifies a pretrained model (for example, an LLM or vision transformer) to better suit a specific, often smaller, dataset by adjusting only a small, low-rank subset of the model's parameters.",
    "og:image": "https://substackcdn.com/image/fetch/$s_!LX62!,w_1200,h_675,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F65c4cc25-a71e-4999-8bcf-a00a850de9a0_1510x876.png",
    "og:url": "https://magazine.sebastianraschka.com/p/lora-and-dora-from-scratch",
    "twitter:title": "Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch",
    "twitter:description": "Low-rank adaptation (LoRA) is a machine learning technique that modifies a pretrained model (for example, an LLM or vision transformer) to better suit a specific, often smaller, dataset by adjusting only a small, low-rank subset of the model's parameters.",
    "twitter:image": "https://substackcdn.com/image/fetch/$s_!1_So!,f_auto,q_auto:best,fl_progressive:steep/https%3A%2F%2Fsebastianraschka.substack.com%2Fapi%2Fv1%2Fpost_preview%2F141797214%2Ftwitter.jpg%3Fversion%3D4",
    "twitter:card": "summary_large_image",
    "article:modified_time": "2025-02-09T12:41:43.737Z"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/9d0f12f4323c/code/body-000.txt",
      "chars": 435,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-001.txt",
      "chars": 312,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-002.txt",
      "chars": 109,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-003.txt",
      "chars": 69,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-004.txt",
      "chars": 92,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-005.txt",
      "chars": 65,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-006.txt",
      "chars": 517,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-007.txt",
      "chars": 98,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-008.txt",
      "chars": 65,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-009.txt",
      "chars": 639,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-010.txt",
      "chars": 268,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-011.txt",
      "chars": 214,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-012.txt",
      "chars": 460,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-013.txt",
      "chars": 436,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-014.txt",
      "chars": 305,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-015.txt",
      "chars": 789,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-016.txt",
      "chars": 232,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-017.txt",
      "chars": 478,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-018.txt",
      "chars": 116,
      "language": "text"
    },
    {
      "file": "attachments/9d0f12f4323c/code/body-019.txt",
      "chars": 356,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!6HOU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3648f6eb-ddbf-4201-9cad-454789f35509_1370x874.png",
      "bytes": 60101,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-fbf2cb745ef2718020fb.png",
      "src": "https://substackcdn.com/image/fetch/$s_!6HOU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3648f6eb-ddbf-4201-9cad-454789f35509_1370x874.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!Fk3V!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee7f7d37-3c0a-4f4a-9244-f73287af6211_1456x612.jpeg",
      "bytes": 59992,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-e60bade87b61e0431d0e.jpeg",
      "src": "https://substackcdn.com/image/fetch/$s_!Fk3V!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee7f7d37-3c0a-4f4a-9244-f73287af6211_1456x612.jpeg",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!wMom!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa11d59f4-a948-42de-a791-114d719f08f1_566x680.png",
      "bytes": 8631,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-3f7ca5f2a05a608ee628.png",
      "src": "https://substackcdn.com/image/fetch/$s_!wMom!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa11d59f4-a948-42de-a791-114d719f08f1_566x680.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!FOLT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd61c168-2fd6-4b2f-b1ef-50e4da5a63e3_652x676.png",
      "bytes": 14448,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-5e9c72acec4e5502680c.png",
      "src": "https://substackcdn.com/image/fetch/$s_!FOLT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd61c168-2fd6-4b2f-b1ef-50e4da5a63e3_652x676.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!866i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24331b05-95b4-4ea0-aa96-d035a929a932_632x926.png",
      "bytes": 86703,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-210b7a15878618f8a39b.png",
      "src": "https://substackcdn.com/image/fetch/$s_!866i!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24331b05-95b4-4ea0-aa96-d035a929a932_632x926.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!pv7G!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6a5bf344-2c27-4197-839f-6c28af722e0f_766x614.png",
      "bytes": 38049,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-a6b7d79efb38a34caa64.png",
      "src": "https://substackcdn.com/image/fetch/$s_!pv7G!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6a5bf344-2c27-4197-839f-6c28af722e0f_766x614.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!aVuN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd07991e9-b89a-44f9-bba3-03bb7cb41288_1370x800.png",
      "bytes": 68862,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-00dd149a5a5802af09e5.png",
      "src": "https://substackcdn.com/image/fetch/$s_!aVuN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd07991e9-b89a-44f9-bba3-03bb7cb41288_1370x800.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!LV-y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa273632c-9434-43d1-9156-fb6d58a1a26e_1600x1137.png",
      "bytes": 374926,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-504aac506c4cdd1cd6bc.png",
      "src": "https://substackcdn.com/image/fetch/$s_!LV-y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa273632c-9434-43d1-9156-fb6d58a1a26e_1600x1137.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!DDex!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27c269b8-e0cc-4be0-a5d3-bd2a79ac7f29_1600x368.png",
      "bytes": 173575,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-3cdc4b8e3f8373563430.png",
      "src": "https://substackcdn.com/image/fetch/$s_!DDex!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27c269b8-e0cc-4be0-a5d3-bd2a79ac7f29_1600x368.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!ejBJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24fbe197-dbbe-4bd5-b25f-59970b19ad44_1444x910.png",
      "bytes": 65694,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-212aefc8f27502b3a883.png",
      "src": "https://substackcdn.com/image/fetch/$s_!ejBJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24fbe197-dbbe-4bd5-b25f-59970b19ad44_1444x910.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!CpnG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3550b9-14a6-4a1d-844d-d72c1bfceed3_1600x488.png",
      "bytes": 405541,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-2cec748a7282135f9c78.png",
      "src": "https://substackcdn.com/image/fetch/$s_!CpnG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3550b9-14a6-4a1d-844d-d72c1bfceed3_1600x488.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!3qw2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F719ec33a-00f2-4542-a539-e3905574621e_1568x1146.png",
      "bytes": 218827,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-30ed57f7d93d4ace943d.png",
      "src": "https://substackcdn.com/image/fetch/$s_!3qw2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F719ec33a-00f2-4542-a539-e3905574621e_1568x1146.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png",
      "bytes": 121470,
      "reused": false,
      "local_path": "attachments/9d0f12f4323c/images/body-1c343ecc9eadd930f4bd.png",
      "src": "https://substackcdn.com/image/fetch/$s_!woQp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://arxiv.org/abs/2402.09353",
      "label": "DoRA: Weight-Decomposed Low-Rank Adaptation",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!6HOU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3648f6eb-ddbf-4201-9cad-454789f35509_1370x874.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2106.09685",
      "label": "LoRA",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!Fk3V!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fee7f7d37-3c0a-4f4a-9244-f73287af6211_1456x612.jpeg",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!wMom!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa11d59f4-a948-42de-a791-114d719f08f1_566x680.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!FOLT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd61c168-2fd6-4b2f-b1ef-50e4da5a63e3_652x676.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!866i!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24331b05-95b4-4ea0-aa96-d035a929a932_632x926.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/rasbt/dora-from-scratch",
      "label": "https://github.com/rasbt/dora-from-scratch",
      "type": "a"
    },
    {
      "href": "https://lightning.ai/lightning-ai/studios/code-lora-from-scratch",
      "label": "LoRA From Scratch – Implement Low-Rank Adaptation for LLMs in PyTorch",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!pv7G!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6a5bf344-2c27-4197-839f-6c28af722e0f_766x614.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!aVuN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd07991e9-b89a-44f9-bba3-03bb7cb41288_1370x800.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1602.07868",
      "label": "Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks paper",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!LV-y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa273632c-9434-43d1-9156-fb6d58a1a26e_1600x1137.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms",
      "label": "Practical Tips for Finetuning LLMs Using LoRA (Low-Rank Adaptation",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!ejBJ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24fbe197-dbbe-4bd5-b25f-59970b19ad44_1444x910.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://github.com/catid/dora/blob/main/dora.py",
      "label": "here",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!CpnG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3d3550b9-14a6-4a1d-844d-d72c1bfceed3_1600x488.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!3qw2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F719ec33a-00f2-4542-a539-e3905574621e_1568x1146.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://amzn.to/4fqvn0D",
      "label": "Build a Large Language Model (From Scratch) book",
      "type": "a"
    },
    {
      "href": "https://substackcdn.com/image/fetch/$s_!woQp!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea1152a0-18d9-4a8a-9398-c6b1ca67726a_1600x900.png",
      "label": "",
      "type": "a"
    },
    {
      "href": "https://www.amazon.com/Build-Large-Language-Model-Scratch/dp/1633437167",
      "label": "brief review",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:46:54Z",
  "source_url": "https://magazine.sebastianraschka.com/p/lora-and-dora-from-scratch",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2402.09353",
      "bytes": 7830644,
      "reused": false,
      "local_path": "papers/repair/bb8e9ba71310fede3329.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2106.09685",
      "bytes": 1609513,
      "reused": false,
      "local_path": "papers/repair/e7d2fed957472d7677ec.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1602.07868",
      "bytes": 315694,
      "reused": false,
      "local_path": "papers/repair/fb85be717b34938f501f.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 23123,
  "expected_body_images": 13,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/9d0f12f4323c/pages/source.html",
  "local_html": "attachments/9d0f12f4323c/pages/article.html",
  "body_base_path": "attachments/9d0f12f4323c/pages/",
  "body_markdown_path": "attachments/9d0f12f4323c/pages/article.md"
}