{ "source_file": "0930", "source_platform": "个人博客", "author": "Lilian Weng / Lil'Log", "title_listed": "Large Transformer Model Inference Optimization", "keywords": "推理优化,量化,蒸馏,并行", "domain": "大模型算法", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/", "publish_date": "2023-01", "corpus_stage": "推理部署", "notes": "大模型推理优化综述，涵盖量化、蒸馏、并行等", "id": "4dab64a7e3ce", "collected_at": "2026-10-02T14:12:45", "title": "Large Transformer Model Inference Optimization | Lil'Log", "body_markdown": "[Updated on 2023-01-24: add a small section on\nDistillation\n.]\nLarge transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale.\nWhy is it hard to run inference for large transformer models?\nBesides the increasing size of SoTA models, there are two main factors contributing to the inference challenge (\nPope et al. 2022\n):\nLarge memory footprint\n. Both model parameters and intermediate states are needed in memory at inference time. For example,\nThe KV cache should be stored in memory during decoding time; E.g. For a batch size of 512 and context length of 2048, the KV cache totals 3TB, that is 3x the model size (!).\nInference cost from the attention mechanism scales quadratically with input sequence length.\nLow parallelizability.\nInference generation is executed in an autoregressive fashion, making the decoding process hard to parallel.\nIn this post, we will look into several approaches for making transformer inference more efficient. Some are general network compression methods, while others are specific to transformer architecture.\nMethods Overview\n#\nWe in general consider the following as goals for model inference optimization:\nReduce the memory footprint of the model by using fewer GPU devices and less GPU memory;\nReduce the desired computation complexity by lowering the number of FLOPs needed;\nReduce the inference latency and make things run faster.\nSeveral methods can be used to make inference cheaper in memory or/and faster in time.\nApply various\nparallelism\nto scale up the model across a large number of GPUs. Smart parallelism of model components and data makes it possible to run a model of trillions of parameters.\nMemory\noffloading\nto offload temporarily unused data to the CPU and read them back when needed later. This helps with memory usage but causes higher latency.\nSmart batching strategy; E.g.\nEffectiveTransformer\npacks consecutive sequences together to remove padding within one batch.\nNetwork\ncompression\ntechniques, such as\npruning, quantization, distillation\n. A model of smaller size, in terms of parameter count or bitwidth, should demand less memory and run faster.\nImprovement specific to a target model architecture. Many\narchitectural changes\n, especially those for attention layers, help with transformer decoding speed.\nCheck\nthe previous post on large model training\non different types of training parallelism and memory saving designs including CPU memory offloading. This post focuses on network compression techniques and architecture-specific improvement for transformer models.\nDistillation\n#\nKnowledge Distillation\n(\nKD\n;\nHinton et al. 2015\n,\nGou et al. 2020\n) is a straightforward way to build a smaller, cheaper model (\n“student model”\n) to speed up inference by transferring skills from a pre-trained expensive model (\n“teacher model”\n) into the student. There is no much restriction on how the student architecture should be constructed, except for a matched output space with the teacher in order to construct a proper learning objective.\nThe generic framework of teacher-student knowledge distillation training. (Image source:\nGou et al. 2020\n)\nGiven a dataset, a student model is trained to mimic outputs of a teacher via distillation loss. Usually a neural network has a softmax layer; For example, a LLM outputs a probability distribution over tokens. Let’s denote the logits layer right before softmax as $\\mathbf{z}_t$ and $\\mathbf{z}_s$ for teacher and student models, respectively. The\ndistillation loss\nminimizes the difference between two softmax outputs with a high temperature $T$. When ground truth labels $\\mathbf{y}$ are known, we can combine it with a\nsupervised\nlearning objective between ground truth and the student’s soft logits using e.g. cross-entropy.\n$$\n\\mathcal{L}_\\text{KD} = \\mathcal{L}_\\text{distll}(\\text{softmax}(\\mathbf{z}_t, T), \\text{softmax}(\\mathbf{z}_s, T)) + \\lambda\\mathcal{L}_\\text{CE}(\\mathbf{y}, \\mathbf{z}_s)\n$$\nwhere $\\lambda$ is a hyperparameter to balance between soft and hard learning objectives. A common choice for $\\mathcal{L}_\\text{distll}$ is KL divergence / cross entropy.\nA successful early trial is\nDistilBERT\n(\nSanh et al. 2019\n) that is able to reduce the parameters of a BERT by 40% while maintaining 97% performance of BERT on fine-tuned downstream tasks and running 71% faster. The loss of pre-training DistilBERT is a combination of soft distillation loss, supervised training loss (i.e.\nMasked language modeling loss\n$\\mathcal{L}_\\text{MLM}$ in the case of BERT) and a special\ncosine embedding loss\nto align the hidden state vectors between teacher and student.\nDistillation can be easily combined with\nquantization\n,\npruning\nor\nsparsification\ntechniques, where the teacher model is the original full-precision, dense model and the student is quantized, pruned, or trimmed to have higher sparsity level.\nQuantization\n#\nThere are two common approaches for applying quantization on a deep neural network:\nPost-Training Quantization (PTQ)\n: A model is first trained to convergence and then we convert its weights to lower precision without more training. It is usually quite cheap to implement, in comparison to training.\nQuantization-Aware Training (QAT)\n: Quantization is applied during pre-training or further fine-tuning. QAT is able to attain better performance but requires extra computation resources and access to representative training data.\nWe should be aware of the gap between theoretical optimal quantization strategy and the hardware kernel support. Due to the lack of GPU kernel support for certain types of matrix multiplication (e.g. INT4 x FP16), not all the methods below result in speedup for the actual inference.\nChallenges for Transformer Quantization\n#\nMany studies on Transformer model quantization have the same observation: A simple low-precision (e.g. 8-bit) post-training quantization leads to significant performance drop mainly due to the high dynamic ranges of activation and a naive activation quantization strategy fails to maintain the capacity.\nOnly quantizing model weights to 8-bit while keeping activation at full precision (`W8A32`) achieves much better results when activations are quantized to 8-bit irrespective of whether weights are in lower precision (`W8A8` and `W32A8`). (Image source:\nBondarenko et al. 2021\n)\nBondarenko et al. (2021)\nobserved in a small BERT model that FFN’s input and output have very different dynamic ranges due to strong outliers in the output tensor. Therefore per-tensor quantization for the FFN’s residual sum is likely to cause a notable error.\nAs the model size continues to grow to billions of parameters, outlier features of high magnitude start to emerge in\nall\ntransformer layers, causing failure of simple low-bit quantization.\nDettmers et al. (2022)\nobserved such a phenomenon for\nOPT\nmodels larger than 6.7B parameters. Larger models have more layers with extreme outliers and these outlier features have a significant impact on the model performance. The scale of activation outliers in a few dimensions can be ~100× larger than most of the other values.\nThe mean zero-shot accuracy over a set of language tasks (WinoGrande, HellaSwag, PIQA, LAMBADA) of OPT models of increasing sizes. (Image source:\nDettmers et al. 2022\n)\nPost-training quantization (PTQ)\n#\nMixed-precision quantization\n#\nThe most straightforward approach for resolving the above quantization challenge is to implement quantization at different precision for weights vs activation.\nGOBO (\nZadeh et al. 2020\n) is one of the first models to apply post-training quantization on transformers (i.e. a small BERT model). It assumes that model weights of each layer follow a Gaussian distribution and therefore detects outliers by tracking mean and standard deviation per layer. Outlier features remain in original form, while other values are split into multiple bins and only corresponding bin indices of weights and the centroid values are stored.\nBased on the observation that only certain activation layers (e.g. residual connections after FFN) in BERT cause big performance drop,\nBondarenko et al. (2021)\nadopted mixed-precision quantization by using 16-bit quantization on problematic activations but 8-bit on others.\nMixed-precision quantization in\nLLM.int8()\n(\nDettmers et al. 2022\n) is implemented via two mixed-precision decompositions:\nBecause matrix multiplication contains a set of independent inner products between row and column vectors, we can impose independent quantization per inner product: Each row and column are scaled by the absolution maximum values and then quantized to INT8.\nOutlier activation features (e.g. 20x larger than other dimensions) remain in FP16 but they represent only a tiny fraction of total weights. How to identify outliers is empirical.\nTwo mixed-precision decompositions of `LLM.int8()`. (Image source:\nDettmers et al. 2022\n)\nQuantization at fine-grained granularity\n#\nComparison of quantization at different granularity. $d$ is the model size / hidden state dimension and $h$ is the number of heads in one MHSA (multi-head self-attention) component.\nNaively quantizing the entire weight matrix in one layer (“per-tensor” or “per-layer” quantization) is easiest to implement but does not lead to good granularity of quantization.\nQ-BERT\n(\nShen, Dong & Ye, et al. 2020\n) applied\ngroup-wise quantization\nto a fine-tuned BERT model, treating an individual matrix $W$ with respect to\neach head\nin MHSA (multi-head self-attention) as one group and then applies Hessian based mixed precision quantization.\nPer-embedding group (PEG)\nactivation quantization was motivated by the observation that outlier values only appear in a few out of $d$ (hidden state / model size) dimensions (\nBondarenko et al. 2021\n). Per-embedding is pretty computationally expensive. In comparison, PEG quantization splits the activation tensor into several evenly sized groups along the embedding dimension where elements in the same group share quantization parameters. To ensure all outliers are grouped together, they apply a deterministic range-based permutation of embedding dimensions, where dimensions are sorted by their value ranges.\nZeroQuant\n(\nYao et al. 2022\n) uses\ngroup-wise quantization\nfor weights, same as in Q-BERT, and\ntoken-wise quantization\nfor activation. To avoid expensive quantization and de-quantization computation, ZeroQuant built customized\nkernel\nto\nfuse\nquantization operation with its previous operator.\nSecond order information for quantization\n#\nQ-BERT (\nShen, Dong & Ye, et al. 2020\n) developed Hessian AWare Quantization (HAWQ) for its mixed-precision quantization. The motivation is that parameters with higher Hessian spectrum (i.e., larger top eigenvalues) are more sensitive to quantization and thus require higher precision. It is essentially a way to identify outliers.\nIn another viewpoint, the problem of quantization is an optimization problem. Given a weight matrix $\\mathbf{W}$ and an input matrix $\\mathbf{X}$ , we want to find a quantized weight matrix $\\hat{\\mathbf{W}}$ to minimize the MSE:\n$$\n\\hat{\\mathbf{W}}^* = {\\arg\\min}_{\\hat{\\mathbf{W}}} | \\mathbf{W}\\mathbf{X} - \\hat{\\mathbf{W}}\\mathbf{X}|\n$$\nGPTQ\n(\nFrantar et al. 2022\n) treats the weight matrix $\\mathbf{W}$ as a collection of row vectors ${\\mathbf{w}}$ and applies quantization to each row independently. GPTQ iteratively quantizes more weights that are selected greedily to minimize the quantization error. The update on selected weights has a closed-form formula, utilizing Hessian matrices. Read more details in the paper and the OBQ (Optimal Brain Quantization;\nFrantar & Alistarh 2022\n) method if interested. GPTQ can reduce the bitwidth of weights in OPT-175B down to 3 or 4 bits without much performance loss, but it only applies to model weights not activation.\nOutlier smoothing\n#\nIt is known that activations are harder to quantize than weights in transformer models.\nSmoothQuant\n(\nXiao & Lin 2022\n) proposed a smart solution to smooth outlier features from activations to weights via mathematically equivalent transformation and then enable quantization on both weights and activations (\nW8A8\n). Because of this, SmoothQuant has better hardware efficiency than mixed-precision quantization.\nSmoothQuant migrates the scale variance from activations to weights offline to reduce the difficulty of activation quantization. Both the resulting new weight and activation matrices are easy to quantize. (Image source:\nXiao & Lin 2022\n)\nConsidering a per-channel smooth factor $\\mathbf{s}$, SmoothQuant scales the weights according to:\n$$\n\\mathbf{Y} = (\\mathbf{X} \\text{diag}(\\mathbf{s})^{-1}) \\cdot (\\text{diag}(\\mathbf{s})\\mathbf{W}) = \\hat{\\mathbf{X}}\\hat{\\mathbf{W}}\n$$\nThe smoothing factor can be easily fused into previous layers’ parameters offline. A hyperparameter $\\alpha$ controls how much we migrate the quantization difficulty from activations to weights: $\\mathbf{s} = \\max (\\vert \\mathbf{X}_j \\vert)^\\alpha / \\max( \\vert \\mathbf{W}_j \\vert )^{1-\\alpha}$. The paper found that $\\alpha=0.5$ is a sweet spot for many LLMs in the experiments. For models with more significant outliers in activation, $\\alpha$ can be adjusted to be larger.\nQuantization-aware training (QAT)\n#\nQuantization-aware training fuses the quantization operation into the pre-training or fine-tuning process. It learns model weights in low-bit representation directly and leads to better performance at the cost of additional training time and computation.\nThe most straightforward approach is to\nfine-tune\nthe model after quantization on a training dataset that is the same as or representative of the pre-training dataset. The training objective can be the same as the one for pre-training (e.g. NLL/MLM in general language model training) or specific to a downstream task that we care about (e.g. Cross entropy for classification).\nAnother approach is to consider the full-precision model as the teacher and the lower-precision model as the student, and then optimize the low-precision model with\ndistillation\nloss. Distillation usually doesn’t need to use the original dataset; E.g. Wikipedia dataset is a good choice and even random tokens can give decent performance gain. The\nLayer-by-layer Knowledge Distillation\n(\nLKD\n;\nYao et al. 2022\n) method quantizes the network layer by layer and uses its original, unquantized version as the teacher model. Given the same inputs, LKD minimizes the MSE between the multiplication with layer weights and the multiplication of quantized layer weights.\nPruning\n#\nNetwork pruning is to reduce the model size by trimming unimportant model weights or connections while the model capacity remains. It may or may not require re-training. Pruning can be\nunstructured\nor\nstructured\n.\nUnstructured pruning\nis allowed to drop any weight or connection, so it does not retain the original network architecture. Unstructured pruning often does not work well with modern hardware and doesn’t lead to actual inference speedup.\nStructured pruning\naims to maintain the dense matrix multiplication form where some elements are zeros. They may need to follow certain pattern restrictions to work with what hardware kernel supports. Here we focus on structured pruning to achieve\nhigh sparsity\nin transformer models.\nA routine workflow to construct a pruned network has three steps:\nTrain a dense network until convergence;\nPrune the network to remove unwanted structure;\nOptionally retrain the network to recover the performance with new weights.\nThe idea of discovering a sparse structure within a dense model via network pruning while the sparse network can still maintain similar performance is motivated by\nLottery Ticket Hypothesis\n(\nLTH\n): A randomly initialized, dense, feed-forward network contains a pool of subnetworks and among them only a subset (a sparse network) are\n“winning tickets”\nwhich can achieve the optimal performance when trained in isolation.\nHow to prune?\n#\nMagnitude pruning\nis simplest yet quite effective pruning method - weights with smallest absolute values are trimmed. In fact, some studies (\nGale et al. 2019\n) found that\nsimple magnitude pruning approaches can achieve comparable or better results than complicated pruning methods\n, such as variational dropout (\nMolchanov et al. 2017\n) and $l_0$ regularization (\nLouizos et al. 2017\n). Magnitude pruning is simple to apply to large models and achieves reasonably consistent performance across a wide range of hyperparameters.\nZhu & Gupta (2017)\nfound that\nlarge sparse models were able to achieve better performance than their small but dense counterparts\n. They proposed\nGradual Magnitude Pruning (GMP)\nalgorithm that increases the sparsity of a network gradually over the course of training. At each training step, weights with smallest absolute values are masked to be zeros to achieve a desired sparsity level $s$ and masked weights do not get gradient update during back-propagation. The desired sparsity level $s$ goes up with more training steps. The process of GMP is sensitive to the learning rate schedule, which should be higher than what’s used in dense network training, but not too high to prevent convergence.\nIterative pruning\n(\nRenda et al. 2020\n) iterates step 2 (prune) & step 3 (retrain) multiple times: Only a small fraction of weights are pruned and the model is retrained in each iteration. The process repeats until a desired sparsity level is reached.\nHow to retrain?\n#\nThe retraining step can be simple fine-tuning using the same pre-training data or other task-specific datasets.\nLottery Ticket Hypothesis\nproposed a\nweight rewinding\nretraining technique: After pruning, the unpruned weights are\nreinitialized back to original values\nearlier in the training and then retrain with the same learning rate schedule.\nLearning rate rewinding\n(\nRenda et al. 2020\n) only resets the learning rate back to its early value, while the unpruned weights stay unchanged since the end of the last train stage. They observed that (1) retraining with weight rewinding outperforms retraining with fine-tuning across networks and datasets and (2) learning rate rewinding matches or outperforms weight rewinding in all tested scenarios.\nSparsity\n#\nSparsity is an effective way to scale up model capacity while keeping model inference computationally efficient. Here we consider two types of sparsity for transformers:\nSparsified dense layers, including both self-attention and FFN layers.\nSparse model architecture; i.e. via incorporating the Mixture-of-Experts (MoE) component.\nN:M Sparsity via Pruning\n#\nN:M sparsity\nis a structured sparsity pattern that works well with modern GPU hardware optimization, in which $N$ out of every $M$ consecutive elements are zeros. For example, the sparse tensor core of Nvidia A100 GPU has support for 2:4 sparsity for faster inference (\nNvidia 2020\n).\nA matrix of 2:4 structured sparsity and its compressed representation. (Image source:\nNvidia blog\n)\nTo sparsify a dense neural network to follow a N:M structured sparsity pattern,\nNvidia (2020)\nsuggested using the three-step\nroutine workflow\nfor training a pruned network: train –> prune to satisfy 2:4 sparsity –> retrain.\nPermuting columns can provide more options in the pruning process to maintain parameters of large magnitude or to satisfy a special restriction like N:M sparsity (\nPool & Yu 2021\n). As long as paired axes of two matrices are permuted in the same order, the results of matrix multiplication would not change. For example,\n(1) Within the self-attention module, if the same permutation order is applied on the axis 1 of query embedding matrix $\\mathbf{Q}$ and the axis 0 of key embedding matrix $\\mathbf{K}^\\top$, the final result of matrix multiplication of $\\mathbf{Q}\\mathbf{K}^\\top$ would stay the same.\nIllustration of same permutation on $\\mathbf{Q}$ (axis 1) and $\\mathbf{K}^\\top$ (axis 0) to keep the results of a self-attention module unchanged.\n(2) Within the FFN layer that contains two MLP layers and one ReLU non-linear layer, we can permute the first linear weight matrix $\\mathbf{W}_1$ along the axis 1 and the second linear weight matrix $\\mathbf{W}_2$ along the axis 0 in the same order.\nIllustration of the same permutation on $\\mathbf{W}_1$ (axis 1) and $\\mathbf{W}_2$ (axis 0) to keep the FFN layer's output unchanged. For simplicity, the bias terms are skipped but the same permutation should be applied on them too.\nTo enforce N:M structured sparsity, let’s split the columns of one matrix into multiple slides of $M$ columns (named “stripe”) and we can easily observe that both the order of columns within each stripe and the order of stripes have no effect on the N:M sparsity restriction.\nPool & Yu (2021)\nproposed an iterative greedy algorithm to find optimal permutation that maximizes the weight magnitude for N:M sparsity. All pairs of channels are speculatively swapped and only the swap that leads to the greatest increase in magnitude is adopted, generating a new permutation and concluding a single iteration. Greedy algorithm may only find local minima, so they introduced two techniques to escape local minima:\nBounded regressions: In practice two random channels are swapped, up to a fixed number of times. The solution search is limited to a depth of only one channel swap to keep the search space broad and shallow.\nNarrow, deep search: Choose multiple stripes and optimize them at the same time.\nAlgorithm of finding the best permutation for N:M sparsity greedily and iteratively. (Image source:\nPool & Yu 2021\n)\nThe network can achieve better performance if it was permuted before pruning, compared to pruning the network in its default channel order.\nTo train a model with N:M sparsity from scratch,\nZhou & Ma, et al. (2021)\nextended STE (Straight-Through Estimator;\nBengio et al. 2013\n), which is commonly used for back-propagation update in model quantization, to work for magnitude pruning and sparse parameter update.\nSTE computes the gradients of dense parameters wrt the pruned network $\\widetilde{W}$, $\\partial \\mathcal{L}/\\partial \\widetilde{W}$, and applies that to the dense network $W$ as an approximation:\n$$\nW_{t+1} \\gets W_t - \\gamma \\frac{\\partial\\mathcal{L}}{\\partial\\widetilde{W}}\n$$\nThe extended version,\nSR-STE\n(Sparse-refined STE), updates the dense weights $W$ by:\n$$\nW_{t+1} \\gets W_t - \\gamma \\frac{\\partial\\mathcal{L}}{\\partial\\widetilde{W}} + \\lambda_W (\\bar{\\mathcal{E}} \\odot W_t)\n$$\nwhere $\\bar{\\mathcal{E}}$ is the mask matrix for $\\widetilde{W}$ and $\\odot$ is element-wise multiplication. SR-STE is proposed to prevent large change in the binary mask by (1) restricting the values of weights pruned in $\\widetilde{W}_t$, and (2) promoting the non-pruned weights in $\\widetilde{W}_t$.\nComparison of STE and SR-STE. $\\odot$ is element-wise product; $\\otimes$ is matrix multiplication. (Image source:\nZhou & Ma, et al. 2021\n)\nDifferent from STE or SR-STE, the\nTop-KAST\n(\nJayakumar et al. 2021\n) method can preserve constant sparsity throughout training in both the forward and backward-passes but does not require forward passes with dense parameters or dense gradients.\nAt one training step $t$, Top-KAST processes as follows:\nSparse forward pass\n: Select a subset of parameters $A^t \\subset \\Theta$, containing top-$K$ parameters by magnitude by each layer, restricted to top $D$-proportion of weights. The parameterization $\\alpha^t$ at time $t$ has parameters zeroed out if it is not in $A^t$ (active weights).\n$$\n\\alpha^t_i = \\begin{cases}\n\\theta^t_i & \\text{ if } i \\in A^t = \\{i \\mid \\theta^t_i \\in \\text{TopK}(\\theta^t, D) \\}\\\\ \n0 & \\text{ otherwise}\n\\end{cases}\n$$\nwhere $\\text{TopK}(\\theta, x)$ selected top $x$ proportion of weights from $\\theta$ based on magnitude.\nSparse backward pass\n: Then apply gradients to a larger parameter subset $B \\subset \\Theta$ where $B$ contains $(D+M)$-proportion of weights and $A \\subset B$. Updating a larger proportion of weights enables more effective exploration of different pruning masks, making it more likely to cause permutations in the top $D$-proportion active weights.\n$$\n\\Delta_{\\theta^t_i} = \\begin{cases}\n-\\eta \\nabla_{\\alpha_t} \\mathcal{L}(y, x, \\alpha^t)_i & \\text{ if } i\\in B^t = \\{i \\mid \\theta^t_i \\in \\text{TopK}(\\theta^t, D+M) \\} \\\\\n0 & \\text{ otherwise }\n\\end{cases}\n$$\nTraining is split into two stages and the additional coordinates in the set $B \\setminus A$ controls how much exploration is brought in. The amount of exploration is expected to diminish gradually through the training process and the mask eventually stabilizes.\nThe pruning mask of Top-KAST stabilizes in time. (Image source:\nJayakumar et al. 2021\n)\nTo prevent rich-get-richer phenomenon, Top-KAST penalizes the magnitude of active weights via a L2 regularization loss to encourage more exploration of new items. Parameters in $B \\setminus A$ are penalized more than $A$ for a higher selection bar during updates to stabilize the mask.\n$$\nL_\\text{penalty}(\\alpha^t_i) = \\begin{cases}\n\\vert \\theta^t_i\\vert & \\text{ if } i \\in A^t \\\\ \n\\vert \\theta^t_i\\vert / D & \\text{ if } i \\in B^t \\setminus A^t \\\\ \n0 & \\text{ otherwise}\n\\end{cases}\n$$\nSparsified Transformer\n#\nScaling Transformer\n(\nJaszczur et al. 2021\n) sparsifies both self-attention and FFN layers in transformer architecture, achieving 37x speedup for single-example inference.\nThe speed of decoding a single token (unbatched inference) by a transformer model when sparsification is applied on different layers. (Image source:\nJaszczur et al. 2021\n)\nSparse FFN layer\n: Each FFN layer contains 2 MLP and one ReLU in-between. Because ReLU will introduce a lot of zeros, they implement a fixed structure on activations to enforce only 1 non-zero value in one block of $N$ elements. The sparsity pattern is dynamic, different for each token.\n$$\n\\begin{aligned}\nY_\\text{sparse} &= \\max(0, xW_1 + b_1) \\odot \\text{Controller}(x) \\\\\n\\text{SparseFFN}(x) &= Y_\\text{sparse} W_2 + b_2 \\\\\n\\text{Controller}(x) &= \\arg\\max(\\text{Reshape}(x C_1 C_2, (-1, N)))\n\\end{aligned}\n$$\nwhere each activation in $Y_\\text{sparse}$ corresponds to one column in $W_1$ and one row in $W_2$. The controller is implemented as a low-rank bottleneck dense layer, $C_1 \\in \\mathbb{R}^{d_\\text{model} \\times d_\\text{lowrank}}, C_2 \\in \\mathbb{R}^{d_\\text{lowrank} \\times d_\\text{ff}}$ and $d_\\text{lowrank} = d_\\text{model} / N$. It uses $\\arg\\max$ for inference to select which columns should be non-zero and Gumbel-softmax trick (\nJang et al. 2016\n) during training. Because we can compute $\\text{Controller}(x)$ before loading FFN weight matrices, we know which columns will be zeroed out and thus choose\nnot to load\nthem into memory for inference speedup.\n(a) Sparse FFN layer; columns in red are not loaded in memory for faster inference. (b) Sparse FFN controller for 1:4 sparsity. (Image source:\nJaszczur et al. 2021\n) *Lilian's side note*: Fig (a) in the illustration from the paper is actually $Y_\\text{sparse} = \\max\\big(0, (xW_1 + b_1) \\odot \\text{Controller}(x)\\big)$, but it doesn't change the results.\nSparse QKV (attention) layer\n: In the attention layer, the dimensionality $d_\\text{model}$ is divided into $S$ modules, each of size $M=d_\\text{model} /S$. To make sure each subdivision can access any part of the embedding, Scaling Transformer introduces a multiplicative layer (i.e., a multiplication layer multiplies inputs from multiple neural network layers element-wise) which can represent arbitrary permutation but contains fewer parameters than a dense layer.\nGiven an input vector $x \\in \\mathbb{R}^{d_\\text{model}}$, the multiplicative layer outputs $y \\in \\mathbb{R}^{S \\times M}$:\n$$\ny_{s,m} = \\sum_i x_i D_{i,s} E_{i,m}\n\\quad\\text{where }D \\in \\mathbb{R}^{d_\\text{model} \\times S}, D \\in \\mathbb{R}^{d_\\text{model} \\times M}\n$$\nThe output of the multiplicative layer is a tensor of size $\\in \\mathbb{R}^{\\text{batch size}\\times \\text{length} \\times S \\times M}$. It then gets processed by a two-dimensional convolutional layer, where $\\text{length}$ and $S$ are treated as the height and width of an image. Such a convolution layer further reduces the parameter count and computation time of attention layer.\n(a) A multiplicative layer is introduced to enable partitions to access any part of an embedding. (b) Combination of multiplicative dense layer and 2-D convolutional layer reduces the number of parameters and computation time of the attention layer. (Image source:\nJaszczur et al. 2021\n)\nTo better work with long sequences, Scaling Transformer is further equipped with LSH (locality-sensitive hashing) attention from\nReformer\n(\nKitaev, et al. 2020\n) and FFN block recurrence, resulting in\nTerraformer\n.\nMixture-of-Experts\n#\nMixture-of-experts (MoE) models depend on a collection of “expert” networks and each example only activates a subset of networks to get predictions. The idea originated back to the 1990s (\nJacobs et al. 1991\n) and is strongly related to ensemble methods. For details on how to incorporate MoE module into transformer, please check my\nprevious post on large model training techniques\nand a survey paper on MoE by\nFedus et al. 2022\n.\nWith MoE architecture, only partial parameters are utilized at decoding time and therefore it saves inference cost. The capacity of each expert can be adjusted with a hyperparameter, capacity factor $C$, and the expert capacity is defined as:\n$$\n\\text{Expert capacity} = \\text{round}(C \\cdot k \\cdot \\frac{\\text{total # tokens in one batch}}{\\text{# experts}})\n$$\nwhere top-$k$ experts are selected per token. Larger $C$ leads to higher expert capacity and improved performance but more expensive computationally. When $C>1$, a slack capacity is added; otherwise, when $C<1$, the routing network needs to ignore some tokens.\nRouting Strategy Improvement\n#\nMoE layer has a routing network to assign a subset of experts for each input token. The routing strategy in vanilla MoE models is to route each token toward preferred experts differently as they come up in the natural order. If a token is routed to experts that have reached their capacity, the token would be marked\n“overflowed” and skipped\n.\nV-MoE\n(Vision MoE;\nRiquelme et al. 2021\n) adds MoE layers into ViT (Vision Transformer). It matches the performance of previous SoTA but only requires\nhalf\nof inference compute. V-MoE can be scaled up to 15B parameters. Their experiments used $k=2$, 32 experts and every-2 expert placement (meaning that MoEs are placed in every other layer).\nSince each expert has a limited capacity, some important and informative tokens may have to be discarded if they come up too late in the predefined sequence order (e.g. the order of words in a sentence, or the order of image patches). To avoid such a drawback in the vanilla routing scheme, V-MoE adopts\nBPR (Batch Priority Routing)\nto assign experts to tokens with a high priority score first. BPR computes a priority score (max or sum of top-$k$ router scores) per token before expert assignment and alters the order of tokens accordingly. This guarantees that the expert capacity buffer would be fulfilled with key tokens first.\nHow image patches are discarded according to priority scores when $C < 1$. (Image source:\nRiquelme et al. 2021\n)\nBPR works much better than vanilla routing when $C\\leq 0.5$, where the model starts dropping a significant amount of tokens. It capacitates the model to be competitive with the dense network even at quite low capacities.\nWhen looking into how to interpret image class-expert association, they observed that early MoE layers are more general, while later MoE layers could be specialized for a few image classes.\nTask MoE\n(Task-level Mixture-of-Experts;\nKudugunta et al. 2021\n) takes the task information into consideration and routes tokens at the\ntask\nlevel instead of the word or token level for machine translation. They used MNMT (multilingual neural machine translation) as an example and group translation tasks based on the target language or language pairs.\nToken level routing is dynamic and the routing decision for each token is made disjointly. Hence, at inference time, the server needs to preload all the experts. In comparison, task level routing is\nstatic\ngiven a fixed task, so the inference server for one task only needs to preload $k$ experts (assuming top-$k$ routing). According to their experiments, Task MoE can achieve similar performance gain as token MoE compared to dense model baseline with 2.6x higher peak throughput and 1.6% of the decoder size.\nTask level MoE is essentially to categorize a distribution of tasks according to predefined\nheuristics\nand incorporate such human knowledge into the router. When such heuristics do not exist (e.g. consider a general sentence continuation task), it would not be straightforward how to utilize Task MoE.\nPR-MoE\n(Pyramid residual MoE;\nRajbhandari et al. 2022\n) has each token pass one fixed MLP and one chosen expert. Due to the observation that MoE at later layers is more beneficial, PR-MoE adopts more exports at later layers. DeepSpeed library implements a flexible multi-expert, multi-data parallelism to enable training PR-MoE with different numbers of experts across layers.\nIllustration of PR-MoE architecture in comparison with a standard MoE. (Image source:\nRajbhandari et al. 2022\n)\nKernel Improvement\n#\nExpert networks can be hosted on different devices. However, when the number of GPUs increases, the number of experts per GPU decreases and the communication between experts (“All-to-all”) grows to be more expensive. All-to-all communication between experts across a number of GPUs relies on P2P APIs of NCCL, which cannot saturate the bandwidth of high-speed links (e.g. NVLink, HDR InfiniBand) at a large scale, as individual chunk gets smaller with more nodes used. The existing all-to-all algorithm performs poorly at large scale with a small workload. There are a variety of kernel improvements to enable more efficient MoE computation, such as making all-to-all communication cheaper/faster.\nBoth the\nDeepSpeed\nlibrary (\nRajbhandari et al. 2022\n) and TUTEL (\nHwang et al. 2022\n) implemented a tree-based\nhierarchical all-to-all\nalgorithm, which runs an intra-node all-to-all followed by an inter-node all-to-all. It reduces the communication hops from $O(G)$ to $O(G_\\text{node} + G / G_\\text{node})$, where $G$ is the total number of GPU nodes and $G_\\text{node}$ is the number of GPU cores per node. Although the communication volume is doubled in such implementation, it enables better scaling with small batches at large scale as the bottleneck is on latency instead of communication bandwidth when the batch size is small.\nDynaMoE\n(\nKossmann et al. 2022\n) uses\ndynamic recompilation\nto adapt the computational resources to dynamic workloads among experts. The\nRECOMPILE\nmechanism compiles the computation graph from scratch and only reallocates resources when needed. It measures how many samples are assigned to each expert and adjusts their capacity factors $C$ dynamically, in order to reduce the memory and computation requirements at run time. Based on the observation that sample-expert assignments converge early in training,\nsample assignment caching\nis introduced after convergence and then\nRECOMPILE\nis used to eliminate the dependency between the gating network and experts.\nArchitectural Optimization\n#\nThe survey paper on\nEfficient Transformers\n(\nTay et al. 2020\n) reviewed a collection of new transformer architectures with improvement for better\ncomputational and memory efficiency\n. Strongly recommend a read. You can also check out my post\n“The Transformer Family Version 2.0”\nfor introduction to a diverse set of transformer archiecture improvements in depth, including changes to make the model cheaper to run.\nCategorization of efficient transformer models.\n(Image source:\nTay et al. 2020\n)\nSince the self-attention mechanism has quadratic time and memory complexity and that is the main bottleneck for better transformer decoding efficiency, all the efficient transformer models have applied some form of sparsity to the otherwise dense attention layer. Here only lists a high-level overview, several derived from\nTay et al. 2020\n.\nSparse Attention Patterns\n#\nFixed Patterns\nlimit the field of view for the attention matrix, using predefined, fixed patterns.\nChunk input sequences into fixed blocks, such as\nBlockwise Attention\n;\nImage Transformer\nuses local attention;\nSparse Transformer\nuses strided attention patterns.\nCombined Patterns\nlearn to sort/cluster the input tokens - enabling a more optimal global view of the sequence while maintaining the efficiency benefits of fixed patterns.\nSparse Transformer\ncombines strided and local attention;\nGiven a high dimensional input tensor, instead of applying attention to the flattened version of the input,\nAxial Transformer\napplies multiple attentions, each along a single axis of the input tensor.\nETC, Longformer and Big Bird\ncombines local and global context, as well as strided or random attention.\nLearnable Patterns\nidentify the optimal attention pattern via learning.\nReformer\nclusters tokens into clusters based on hash-based similarity (LSH);\nRouting Transformer\nruns $k$-means clustering on tokens;\nSinkhorn Sorting Network\nlearns to sort blocks of input sequence.\nRecurrence\n#\nRecurrence mechanism connects multiple blocks/segments via recurrence.\nTransformer-XL\nmakes use of longer context by reusing hidden states between segments.\nUniversal Transformer\ncombines self-attention with the recurrent mechanism in RNN.\nCompressive Transformer\nis an extension of Transformer-XL with additional memory, containing a set of memory slots for past activiations and compressive memory slots for compressed activations. Whenever the model accepts a new input segment, the oldest activations in the primary memory are moved to the compressed memory where a compression function is applied.\nMemory Saving Designs\n#\nMemory saving designs refer to changes of the architecture to use less memory.\nLinformer\nprojects the length dimension of keys and values to a lower-dimensional representation ($N \\to k$) and thus the memory complexity is reduced from $N \\times N$ to $N \\times k$.\nShazeer (2019)\nproposed\nmulti-query attention\nwhich has the keys and values shared across different attention “heads”, greatly reducing the size of these tensors and the memory cost.\nRandom feature attention and Performer\nuse\nkernel methods\nto achieve a cheaper mathematical format of the self-attention mechanism.\nAdaptive Attention\n#\nAdaptive attention\nenables the model to learn the optimal attention span or decide on when to do early exiting for different input tokens.\nAdaptive Attention Span\ntrains the model to learn the optimal attention span per token per head via a soft mask between the token and other keys.\nUniversal Transformer\nincorporates recurrent mechanism and uses\nACT (Adaptive computation time)\nto dynamically decide the number of recurrent steps.\nDepth-Adaptive Transformer and CALM\nlearns when to early exit the computation layers per token using some confidence measures to achieve good performance-efficiency tradeoffs.\nCitation\n#\nCited as:\nWeng, Lilian. (Jan 2023). Large Transformer Model Inference Optimization. Lil’Log. https://lilianweng.github.io/posts/2023-01-10-inference-optimization/.\nOr\n@article{weng2023inference,\n title = \"Large Transformer Model Inference Optimization\",\n author = \"Weng, Lilian\",\n journal = \"Lil'Log\",\n year = \"2023\",\n month = \"Jan\",\n url = \"https://lilianweng.github.io/posts/2023-01-10-inference-optimization/\"\n}\nReferences\n#\n[1] Bondarenko et al.\n“Understanding and overcoming the challenges of efficient transformer quantization”\nACL 2021.\n[2] Dettmers et al.\n“LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”\nNeuriPS 2022\n[3] Zadeh et al.\n“Gobo: Quantizing attention-based NLP models for low latency and energy efficient inference.”\nMICRO 2020\n[4] Shen, Dong & Ye, et al.\n“Q-BERT: Hessian based ultra low precision quantization of BERT”\nAAAI 2020.\n[5] Yao et al.\n“ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers”\narXiv preprint arXiv:2206.01861 (2022).\n[6] Frantar et al.\n“GPTQ: Accurate Quantization for Generative Pre-trained Transformers”\narXiv preprint arXiv:2210.17323 (2022).\n[7] Xiao & Lin\n“SmoothQuant: Accelerated sparse neural training: A provable and efficient method to find N:M transposable masks.”\narXiv preprint arXiv:2211.10438 (2022). |\ncode\n[8] Pool & Yu.\n“Channel Permutations for N:M Sparsity.”\nNeuriPS 2021. |\ncode\n[9] Zhou & Ma, et al.\n“Learning N:M fine-grained structured sparse neural networks from scratch.”\narXiv preprint arXiv:2102.04010 (2021).\n[10] Jayakumar et al.\n“Top-KAST: Top-K Always Sparse Training.”\nNeuriPS 2020.\n[11] Nvidia.\n“Nvidia A100 tensor core GPU architecture.”\n2020.\n[12] Gale, Elsen & Hooker\n“The State of Sparsity in Deep Neural Networks.”\narXiv preprint arXiv:1902.09574 (2019).\n[13] Zhu & Gupta.\n“To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression.”\narXiv preprint arXiv:1710.01878 (2017).\n[14] Renda et al.\n“Comparing rewinding and fine-tuning in neural network pruning.”\narXiv preprint arXiv:2003.02389 (2020).\n[15] Zhou & Ma, et al.\n“Learning N:M fine-grained structured sparse neural networks from scratch.”\narXiv preprint arXiv:2102.04010 (2021).\n[16] Pool & Yu.\n“Channel Permutations for N:M Sparsity.”\nNeuriPS 2021. |\ncode\n[17] Jaszczur et al.\n“Sparse is Enough in Scaling Transformers.”\nNeuriPS 2021.\n[18] Mishra et al.\n“An Survey of Neural Network Compression.”\narXiv preprint arXiv:1710.09282 (2017).\n[19] Fedus et al.\n“A Review of Sparse Expert Models in Deep Learning.”\narXiv preprint arXiv:2209.01667 (2022)..\n[20] Riquelme et al.\n“Scaling vision with sparse mixture of experts.”\nNeuriPS 2021.\n[21] Kudugunta et al.\n“Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference.”\narXiv preprint arXiv:2110.03742 (2021).\n[22] Rajbhandari et al.\n“DeepSpeed-MoE: Advancing mixture-of-experts inference and training to power next-generation ai scale.”\narXiv preprint arXiv:2201.05596 (2022).\n[23] Kossmann et al.\n“Optimizing mixture of experts using dynamic recompilations.”\narXiv preprint arXiv:2205.01848 (2022).\n[24] Hwang et al.\n“Tutel: Adaptive mixture-of-experts at scale.”\narXiv preprint arXiv:2206.03382 (2022). |\ncode\n[25] Noam Shazeer.\n“Fast Transformer Decoding: One Write-Head is All You Need.”\narXiv preprint arXiv:1911.02150 (2019).\n[26] Tay et al.\n“Efficient Transformers: A Survey.”\nACM Computing Surveys 55.6 (2022): 1-28.\n[27] Pope et al.\n“Efficiently Scaling Transformer Inference.”\narXiv preprint arXiv:2211.05102 (2022).\n[28] Frankle & Carbin.\n“The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks”\nICLR 2019.\n[29] Elabyad et al.\n“Depth-Adaptive Transformer”\nICLR 2020.\n[30] Schuster et al.\n“Confident Adaptive Language Modeling”\narXiv preprint arXiv:2207.07061 (2022).\n[31] Gou et al.\n“https://arxiv.org/abs/2006.05525”\narXiv preprint arXiv:2006.05525 (2020).\n[32] Hinton et al.\n“Distilling the Knowledge in a Neural Network”\nNIPS 2014.\n[33] Sanh et al.\n“DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”\nWorkshop on Energy Efficient Machine Learning and Cognitive Computing @ NeuriPS 2019.", "metadata": { "title": "Large Transformer Model Inference Optimization | Lil'Log", "description": "[Updated on 2023-01-24: add a small section on Distillation.]\nLarge transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale.\nWhy is it hard to run inference for large transformer models? Besides the increasing size of SoTA models, there are two main factors contributing to the inference challenge (Pope et al. 2022):", "keywords": "language-model, long-read\", architecture, attention, transformer, foundation", "author": "Lilian Weng", "og:title": "Large Transformer Model Inference Optimization", "og:description": "[Updated on 2023-01-24: add a small section on Distillation.]\nLarge transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale.\nWhy is it hard to run inference for large transformer models? Besides the increasing size of SoTA models, there are two main factors contributing to the inference challenge (Pope et al. 2022):", "og:type": "article", "og:url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/", "twitter:card": "summary", "twitter:title": "Large Transformer Model Inference Optimization", "twitter:description": "[Updated on 2023-01-24: add a small section on Distillation.]\nLarge transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale.\nWhy is it hard to run inference for large transformer models? Besides the increasing size of SoTA models, there are two main factors contributing to the inference challenge (Pope et al. 2022):", "article:section": "posts", "article:published_time": "2023-01-10T10:00:00-07:00", "article:modified_time": "2023-01-10T10:00:00-07:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/4dab64a7e3ce/code/body-000.txt", "chars": 266, "language": "text" } ], "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/distillation.png", "bytes": 384775, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-d80699a6bc9957bad425.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/distillation.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/quantization-experiment-table.png", "bytes": 92259, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-8561f90b612da84999be.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/quantization-experiment-table.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/OPT-models-outlier.png", "bytes": 309233, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-298555759af3f16fae32.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/OPT-models-outlier.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/LLM-int8.png", "bytes": 269052, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-05c2da1e57e9a43ebd3b.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/LLM-int8.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/quantization-granularity.png", "bytes": 138899, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-18a0ab9a2af816934187.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/quantization-granularity.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/SmoothQuant.png", "bytes": 223836, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-265b7c1acfbfdba2321d.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/SmoothQuant.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/2-to-4-sparsity.png", "bytes": 233214, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-331145d4eb48e456ba51.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/2-to-4-sparsity.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/permutation-QK.png", "bytes": 161048, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-64ed566037e1b1a18f1a.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/permutation-QK.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/permutation-FFN.png", "bytes": 323173, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-6a9644ce3f0ca9e558ce.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/permutation-FFN.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/N-to-M-sparsity-permutation-algo.png", "bytes": 229591, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-1f8f554e117b9c2d9012.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/N-to-M-sparsity-permutation-algo.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/SR-STE.png", "bytes": 220405, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-381d94847a642e9deceb.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/SR-STE.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/Top-KAST-stabilize.png", "bytes": 443818, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-ab3de7379d4e1e3d6cb9.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/Top-KAST-stabilize.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/scaling-transformer-speedup-table.png", "bytes": 162803, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-4137551652ce9cf07c12.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/scaling-transformer-speedup-table.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/sparse-FFN.png", "bytes": 174385, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-9a8a4982b73fd9d05b58.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/sparse-FFN.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/sparse-QKV.png", "bytes": 209687, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-219ebdabc4785ae2a877.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/sparse-QKV.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/BPR.png", "bytes": 464137, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-0dee3b14ad07c424a838.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/BPR.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/PR-MoE.png", "bytes": 419098, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-4f389df2cd4743bdfc44.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/PR-MoE.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/efficient-transformer.png", "bytes": 590730, "reused": false, "local_path": "attachments/4dab64a7e3ce/images/body-78c0246c48c2b2d83b4d.png", "src": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/efficient-transformer.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://arxiv.org/abs/2211.05102", "label": "Pope et al. 2022", "type": "a" }, { "href": "https://github.com/bytedance/effective_transformer", "label": "EffectiveTransformer", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2021-09-25-train-large/", "label": "the previous post on large model training", "type": "a" }, { "href": "https://arxiv.org/abs/1503.02531", "label": "Hinton et al. 2015", "type": "a" }, { "href": "https://arxiv.org/abs/2006.05525", "label": "Gou et al. 2020", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/”https:/arxiv.org/abs/2006.05525”", "label": "Gou et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1910.01108", "label": "Sanh et al. 2019", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2019-01-31-lm/", "label": "Masked language modeling loss", "type": "a" }, { "href": "https://arxiv.org/abs/2109.12948", "label": "Bondarenko et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2208.07339", "label": "Dettmers et al. (2022)", "type": "a" }, { "href": "https://arxiv.org/abs/2205.01068", "label": "OPT", "type": "a" }, { "href": "https://arxiv.org/abs/2005.03842", "label": "Zadeh et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1909.05840", "label": "Shen, Dong & Ye, et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/2206.01861", "label": "Yao et al. 2022", "type": "a" }, { "href": "https://arxiv.org/abs/2210.17323", "label": "Frantar et al. 2022", "type": "a" }, { "href": "https://arxiv.org/abs/2208.11580", "label": "Frantar & Alistarh 2022", "type": "a" }, { "href": "https://arxiv.org/abs/2211.10438", "label": "Xiao & Lin 2022", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2019-03-14-overfit/", "label": "Lottery Ticket Hypothesis", "type": "a" }, { "href": "https://arxiv.org/abs/1902.09574", "label": "Gale et al. 2019", "type": "a" }, { "href": "https://arxiv.org/abs/1701.05369", "label": "Molchanov et al. 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1712.01312", "label": "Louizos et al. 2017", "type": "a" }, { "href": "https://arxiv.org/abs/1710.01878", "label": "Zhu & Gupta (2017)", "type": "a" }, { "href": "https://arxiv.org/abs/2003.02389", "label": "Renda et al. 2020", "type": "a" }, { "href": "https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf", "label": "Nvidia 2020", "type": "a" }, { "href": "https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt/", "label": "Nvidia blog", "type": "a" }, { "href": "https://proceedings.neurips.cc/paper/2021/hash/6e8404c3b93a9527c8db241a1846599a-Abstract.html", "label": "Pool & Yu 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2102.04010", "label": "Zhou & Ma, et al. (2021)", "type": "a" }, { "href": "https://arxiv.org/abs/1308.3432", "label": "Bengio et al. 2013", "type": "a" }, { "href": "https://arxiv.org/abs/2106.03517", "label": "Jayakumar et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2111.12763", "label": "Jaszczur et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/1611.01144", "label": "Jang et al. 2016", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/", "label": "Reformer", "type": "a" }, { "href": "https://arxiv.org/abs/2001.04451", "label": "Kitaev, et al. 2020", "type": "a" }, { "href": "https://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf", "label": "Jacobs et al. 1991", "type": "a" }, { "href": "https://arxiv.org/abs/2209.01667", "label": "Fedus et al. 2022", "type": "a" }, { "href": "https://arxiv.org/abs/2106.05974", "label": "Riquelme et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2110.03742", "label": "Kudugunta et al. 2021", "type": "a" }, { "href": "https://arxiv.org/abs/2201.05596", "label": "Rajbhandari et al. 2022", "type": "a" }, { "href": "https://arxiv.org/abs/2206.03382", "label": "Hwang et al. 2022", "type": "a" }, { "href": "https://arxiv.org/abs/2205.01848", "label": "Kossmann et al. 2022", "type": "a" }, { "href": "https://arxiv.org/abs/2009.06732", "label": "Tay et al. 2020", "type": "a" }, { "href": "https://arxiv.org/abs/1912.12180", "label": "Axial Transformer", "type": "a" }, { "href": "https://arxiv.org/abs/2002.11296", "label": "Sinkhorn Sorting Network", "type": "a" }, { "href": "https://arxiv.org/abs/1911.02150", "label": "Shazeer (2019)", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/(https:/lilianweng.github.io/posts/2022-09-08-ntk/", "label": "kernel methods", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2020-04-07-the-transformer-family/", "label": "ACT (Adaptive computation time)", "type": "a" }, { "href": "https://github.com/mit-han-lab/smoothquant", "label": "code", "type": "a" }, { "href": "https://github.com/NVIDIA/apex/tree/master/apex/contrib/sparsity", "label": "code", "type": "a" }, { "href": "https://arxiv.org/abs/2010.03954", "label": "“An Survey of Neural Network Compression.”", "type": "a" }, { "href": "https://github.com/microsoft/tutel", "label": "code", "type": "a" }, { "href": "https://arxiv.org/abs/1803.03635", "label": "“The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks”", "type": "a" }, { "href": "https://arxiv.org/abs/1910.10073", "label": "“Depth-Adaptive Transformer”", "type": "a" }, { "href": "https://arxiv.org/abs/2207.07061", "label": "“Confident Adaptive Language Modeling”", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:51:12Z", "source_url": "https://lilianweng.github.io/posts/2023-01-10-inference-optimization/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/2211.05102", "bytes": 1036959, "reused": false, "local_path": "papers/repair/015657916cb9c4c1a696.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1503.02531", "bytes": 106630, "reused": true, "local_path": "papers/repair/c71d5138834ccb8392cc.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2006.05525", "bytes": 1327529, "reused": true, "local_path": "papers/repair/f7386fee4f3f74700076.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1910.01108", "bytes": 436085, "reused": false, "local_path": "papers/repair/9d327e2a83cac2e9902e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2109.12948", "bytes": 2545950, "reused": false, "local_path": "papers/repair/5f9170897996ecd4579d.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2208.07339", "bytes": 719021, "reused": true, "local_path": "papers/repair/30fa6cd91bee75957c34.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2205.01068", "bytes": 1765159, "reused": true, "local_path": "papers/repair/ca8091fa24e3679f3dfb.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2005.03842", "bytes": 2418478, "reused": false, "local_path": "papers/repair/d89d0465e957a3a5a705.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1909.05840", "bytes": 2755540, "reused": false, "local_path": "papers/repair/bfcaa2ab7d2ee04d66e8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2206.01861", "bytes": 1018769, "reused": false, "local_path": "papers/repair/645f822a10179dc521f1.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2210.17323", "bytes": 521818, "reused": true, "local_path": "papers/repair/af396e9b1e330cf9d61c.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2208.11580", "bytes": 739753, "reused": false, "local_path": "papers/repair/9dcf0621c547c97a0f4e.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2211.10438", "bytes": 5378996, "reused": false, "local_path": "papers/repair/41f2cea97b2cd592924a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1902.09574", "bytes": 754887, "reused": false, "local_path": "papers/repair/f119c6cc7304c1f4c50f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1701.05369", "bytes": 489840, "reused": false, "local_path": "papers/repair/ce41a50d1f4834aa9be7.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1712.01312", "bytes": 752872, "reused": false, "local_path": "papers/repair/881e2c9b2b45b7a47b65.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1710.01878", "bytes": 370548, "reused": false, "local_path": "papers/repair/9893eb6c284edf737efe.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2003.02389", "bytes": 11905938, "reused": false, "local_path": "papers/repair/d101e1fe7a2ccc3f76d1.pdf" }, { "status": "ok", "url": "https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf", "bytes": 7979890, "reused": false, "local_path": "papers/repair/e5b3e46caca9df149d72.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2102.04010", "bytes": 573436, "reused": false, "local_path": "papers/repair/34ee0270f51d67be202d.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1308.3432", "bytes": 403557, "reused": false, "local_path": "papers/repair/bfbd18a884cceb36422d.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2106.03517", "bytes": 591603, "reused": false, "local_path": "papers/repair/abd6b82b8ae0381306eb.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2111.12763", "bytes": 704231, "reused": false, "local_path": "papers/repair/e19056dab2f62387345c.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1611.01144", "bytes": 1403159, "reused": true, "local_path": "papers/repair/bcb0690cbb2cea7f5ef8.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2001.04451", "bytes": 658515, "reused": true, "local_path": "papers/repair/362c024f67418a537ddf.pdf" }, { "status": "ok", "url": "https://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf", "bytes": 634556, "reused": false, "local_path": "papers/repair/75bf780c8dc18b4c7eee.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2209.01667", "bytes": 3918653, "reused": true, "local_path": "papers/repair/ee0b2579a7395de9f54c.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2106.05974", "bytes": 25383956, "reused": false, "local_path": "papers/repair/ec06b7ba14b6bafc7d28.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2110.03742", "bytes": 2689859, "reused": false, "local_path": "papers/repair/fdae0d14ec7d4eecdf71.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2201.05596", "bytes": 1349543, "reused": false, "local_path": "papers/repair/904b30af36e22fb22fe4.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2206.03382", "bytes": 2175799, "reused": false, "local_path": "papers/repair/961d567585791b712aff.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2205.01848", "bytes": 1792388, "reused": false, "local_path": "papers/repair/89e4731440a58c1b166f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2009.06732", "bytes": 847824, "reused": true, "local_path": "papers/repair/5442c18a5d3ec7c8eca5.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1912.12180", "bytes": 1648508, "reused": false, "local_path": "papers/repair/5ee01954eef86e6cce29.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2002.11296", "bytes": 450138, "reused": false, "local_path": "papers/repair/0008d87e3417535c824a.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1911.02150", "bytes": 142627, "reused": false, "local_path": "papers/repair/9afa6d0d90be5b43897c.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2010.03954", "bytes": 481837, "reused": false, "local_path": "papers/repair/ab3d550b7a22fc1adf4d.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1803.03635", "bytes": 4001475, "reused": false, "local_path": "papers/repair/85eb00976bcf06c5e750.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1910.10073", "bytes": 480824, "reused": true, "local_path": "papers/repair/929832372275e4ca6acd.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/2207.07061", "bytes": 3006455, "reused": true, "local_path": "papers/repair/3e493b8dcc10d3f89436.pdf" } ], "actual_video_files": [], "body_chars": 43450, "expected_body_images": 18, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/4dab64a7e3ce/pages/source.html", "local_html": "attachments/4dab64a7e3ce/pages/article.html", "body_base_path": "attachments/4dab64a7e3ce/pages/", "body_markdown_path": "attachments/4dab64a7e3ce/pages/article.md" }