{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "What are Diffusion Models?",
  "keywords": "扩散模型,DDPM,生成模型",
  "domain": "图像处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/",
  "publish_date": "2021-07",
  "corpus_stage": "预训练/SFT",
  "notes": "扩散模型经典入门，被广泛引用为Diffusion必读材料",
  "id": "93e53a97f2e3",
  "collected_at": "2026-10-02T14:12:39",
  "title": "What are Diffusion Models? | Lil'Log",
  "body_markdown": "[Updated on 2021-09-19: Highly recommend this blog post on\nscore-based generative modeling\nby Yang Song (author of several key papers in the references)].\n[Updated on 2022-08-27: Added\nclassifier-free guidance\n,\nGLIDE\n,\nunCLIP\nand\nImagen\n.\n[Updated on 2022-08-31: Added\nlatent diffusion model\n.\n[Updated on 2024-04-13: Added\nprogressive distillation\n,\nconsistency models\n, and the\nModel Architecture section\n.\nSo far, I’ve written about three types of generative models,\nGAN\n,\nVAE\n, and\nFlow-based\nmodels. They have shown great success in generating high-quality samples, but each has some limitations of its own. GAN models are known for potentially unstable training and less diversity in generation due to their adversarial training nature. VAE relies on a surrogate loss. Flow models have to use specialized architectures to construct reversible transform.\nDiffusion models are inspired by non-equilibrium thermodynamics. They define a Markov chain of diffusion steps to slowly add random noise to data and then learn to reverse the diffusion process to construct desired data samples from the noise. Unlike VAE or flow models, diffusion models are learned with a fixed procedure and the latent variable has high dimensionality (same as the original data).\nOverview of different types of generative models.\nWhat are Diffusion Models?\n#\nSeveral diffusion-based generative models have been proposed with similar ideas underneath, including\ndiffusion probabilistic models\n(\nSohl-Dickstein et al., 2015\n),\nnoise-conditioned score network\n(\nNCSN\n;\nYang & Ermon, 2019\n), and\ndenoising diffusion probabilistic models\n(\nDDPM\n;\nHo et al. 2020\n).\nForward diffusion process\n#\nGiven a data point sampled from a real data distribution $\\mathbf{x}_0 \\sim q(\\mathbf{x})$, let us define a\nforward diffusion process\nin which we add small amount of Gaussian noise to the sample in $T$ steps, producing a sequence of noisy samples $\\mathbf{x}_1, \\dots, \\mathbf{x}_T$. The step sizes are controlled by a variance schedule $\\{\\beta_t \\in (0, 1)\\}_{t=1}^T$.\n$$\nq(\\mathbf{x}_t \\vert \\mathbf{x}_{t-1}) = \\mathcal{N}(\\mathbf{x}_t; \\sqrt{1 - \\beta_t} \\mathbf{x}_{t-1}, \\beta_t\\mathbf{I}) \\quad\nq(\\mathbf{x}_{1:T} \\vert \\mathbf{x}_0) = \\prod^T_{t=1} q(\\mathbf{x}_t \\vert \\mathbf{x}_{t-1})\n$$\nThe data sample $\\mathbf{x}_0$ gradually loses its distinguishable features as the step $t$ becomes larger. Eventually when $T \\to \\infty$, $\\mathbf{x}_T$ is equivalent to an isotropic Gaussian distribution.\nThe Markov chain of forward (reverse) diffusion process of generating a sample by slowly adding (removing) noise. (Image source:\nHo et al. 2020\nwith a few additional annotations)\nA nice property of the above process is that we can sample $\\mathbf{x}_t$ at any arbitrary time step $t$ in a closed form using\nreparameterization trick\n. Let $\\alpha_t = 1 - \\beta_t$ and $\\bar{\\alpha}_t = \\prod_{i=1}^t \\alpha_i$:\n$$\n\\begin{aligned}\n\\mathbf{x}_t \n&= \\sqrt{\\alpha_t}\\mathbf{x}_{t-1} + \\sqrt{1 - \\alpha_t}\\boldsymbol{\\epsilon}_{t-1} & \\text{ ;where } \\boldsymbol{\\epsilon}_{t-1}, \\boldsymbol{\\epsilon}_{t-2}, \\dots \\sim \\mathcal{N}(\\mathbf{0}, \\mathbf{I}) \\\\\n&= \\sqrt{\\alpha_t \\alpha_{t-1}} \\mathbf{x}_{t-2} + \\sqrt{1 - \\alpha_t \\alpha_{t-1}} \\bar{\\boldsymbol{\\epsilon}}_{t-2} & \\text{ ;where } \\bar{\\boldsymbol{\\epsilon}}_{t-2} \\text{ merges two Gaussians (*).} \\\\\n&= \\dots \\\\\n&= \\sqrt{\\bar{\\alpha}_t}\\mathbf{x}_0 + \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon} \\\\\nq(\\mathbf{x}_t \\vert \\mathbf{x}_0) &= \\mathcal{N}(\\mathbf{x}_t; \\sqrt{\\bar{\\alpha}_t} \\mathbf{x}_0, (1 - \\bar{\\alpha}_t)\\mathbf{I})\n\\end{aligned}\n$$\n(*) Recall that when we merge two Gaussians  with different variance, $\\mathcal{N}(\\mathbf{0}, \\sigma_1^2\\mathbf{I})$ and $\\mathcal{N}(\\mathbf{0}, \\sigma_2^2\\mathbf{I})$, the new distribution is $\\mathcal{N}(\\mathbf{0}, (\\sigma_1^2 + \\sigma_2^2)\\mathbf{I})$. Here the merged standard deviation is $\\sqrt{(1 - \\alpha_t) + \\alpha_t (1-\\alpha_{t-1})} = \\sqrt{1 - \\alpha_t\\alpha_{t-1}}$.\nUsually, we can afford a larger update step when the sample gets noisier, so $\\beta_1 < \\beta_2 < \\dots < \\beta_T$ and therefore $\\bar{\\alpha}_1 > \\dots > \\bar{\\alpha}_T$.\nConnection with stochastic gradient Langevin dynamics\n#\nLangevin dynamics is a concept from physics, developed for statistically modeling molecular systems. Combined with stochastic gradient descent,\nstochastic gradient Langevin dynamics\n(\nWelling & Teh 2011\n) can produce samples from a probability density $p(\\mathbf{x})$ using only the gradients $\\nabla_\\mathbf{x} \\log p(\\mathbf{x})$ in a Markov chain of updates:\n$$\n\\mathbf{x}_t = \\mathbf{x}_{t-1} + \\frac{\\delta}{2} \\nabla_\\mathbf{x} \\log p(\\mathbf{x}_{t-1}) + \\sqrt{\\delta} \\boldsymbol{\\epsilon}_t\n,\\quad\\text{where }\n\\boldsymbol{\\epsilon}_t \\sim \\mathcal{N}(\\mathbf{0}, \\mathbf{I})\n$$\nwhere $\\delta$ is the step size. When $T \\to \\infty, \\epsilon \\to 0$, $\\mathbf{x}_T$ equals to the true probability density $p(\\mathbf{x})$.\nCompared to standard SGD, stochastic gradient Langevin dynamics injects Gaussian noise into the parameter updates to avoid collapses into local minima.\nReverse diffusion process\n#\nIf we can reverse the above process and sample from $q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t)$, we will be able to recreate the true sample from a Gaussian noise input, $\\mathbf{x}_T \\sim \\mathcal{N}(\\mathbf{0}, \\mathbf{I})$. Note that if $\\beta_t$ is small enough, $q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t)$ will also be Gaussian. Unfortunately, we cannot easily estimate $q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t)$ because it needs to use the entire dataset and therefore we need to learn a model $p_\\theta$ to approximate these conditional probabilities in order to run the\nreverse diffusion process\n.\n$$\np_\\theta(\\mathbf{x}_{0:T}) = p(\\mathbf{x}_T) \\prod^T_{t=1} p_\\theta(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t) \\quad\np_\\theta(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t) = \\mathcal{N}(\\mathbf{x}_{t-1}; \\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t), \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t))\n$$\nAn example of training a diffusion model for modeling a 2D swiss roll data. (Image source:\nSohl-Dickstein et al., 2015\n)\nIt is noteworthy that the reverse conditional probability is tractable when conditioned on $\\mathbf{x}_0$:\n$$\nq(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0) = \\mathcal{N}(\\mathbf{x}_{t-1}; \\color{blue}{\\tilde{\\boldsymbol{\\mu}}}(\\mathbf{x}_t, \\mathbf{x}_0), \\color{red}{\\tilde{\\beta}_t} \\mathbf{I})\n$$\nUsing Bayes’ rule, we have:\n$$\n\\begin{aligned}\nq(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0) \n&= q(\\mathbf{x}_t \\vert \\mathbf{x}_{t-1}, \\mathbf{x}_0) \\frac{ q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_0) }{ q(\\mathbf{x}_t \\vert \\mathbf{x}_0) } \\\\\n&\\propto \\exp \\Big(-\\frac{1}{2} \\big(\\frac{(\\mathbf{x}_t - \\sqrt{\\alpha_t} \\mathbf{x}_{t-1})^2}{\\beta_t} + \\frac{(\\mathbf{x}_{t-1} - \\sqrt{\\bar{\\alpha}_{t-1}} \\mathbf{x}_0)^2}{1-\\bar{\\alpha}_{t-1}} - \\frac{(\\mathbf{x}_t - \\sqrt{\\bar{\\alpha}_t} \\mathbf{x}_0)^2}{1-\\bar{\\alpha}_t} \\big) \\Big) \\\\\n&= \\exp \\Big(-\\frac{1}{2} \\big(\\frac{\\mathbf{x}_t^2 - 2\\sqrt{\\alpha_t} \\mathbf{x}_t \\color{blue}{\\mathbf{x}_{t-1}} \\color{black}{+ \\alpha_t} \\color{red}{\\mathbf{x}_{t-1}^2} }{\\beta_t} + \\frac{ \\color{red}{\\mathbf{x}_{t-1}^2} \\color{black}{- 2 \\sqrt{\\bar{\\alpha}_{t-1}} \\mathbf{x}_0} \\color{blue}{\\mathbf{x}_{t-1}} \\color{black}{+ \\bar{\\alpha}_{t-1} \\mathbf{x}_0^2}  }{1-\\bar{\\alpha}_{t-1}} - \\frac{(\\mathbf{x}_t - \\sqrt{\\bar{\\alpha}_t} \\mathbf{x}_0)^2}{1-\\bar{\\alpha}_t} \\big) \\Big) \\\\\n&= \\exp\\Big( -\\frac{1}{2} \\big( \\color{red}{(\\frac{\\alpha_t}{\\beta_t} + \\frac{1}{1 - \\bar{\\alpha}_{t-1}})} \\mathbf{x}_{t-1}^2 - \\color{blue}{(\\frac{2\\sqrt{\\alpha_t}}{\\beta_t} \\mathbf{x}_t + \\frac{2\\sqrt{\\bar{\\alpha}_{t-1}}}{1 - \\bar{\\alpha}_{t-1}} \\mathbf{x}_0)} \\mathbf{x}_{t-1} \\color{black}{ + C(\\mathbf{x}_t, \\mathbf{x}_0) \\big) \\Big)}\n\\end{aligned}\n$$\nwhere $C(\\mathbf{x}_t, \\mathbf{x}_0)$ is some function not involving $\\mathbf{x}_{t-1}$ and details are omitted. Following the standard Gaussian density function, the mean and variance can be parameterized as follows (recall that $\\alpha_t = 1 - \\beta_t$ and $\\bar{\\alpha}_t = \\prod_{i=1}^t \\alpha_i$):\n$$\n\\begin{aligned}\n\\tilde{\\beta}_t \n&= 1/(\\frac{\\alpha_t}{\\beta_t} + \\frac{1}{1 - \\bar{\\alpha}_{t-1}}) \n= 1/(\\frac{\\alpha_t - \\bar{\\alpha}_t + \\beta_t}{\\beta_t(1 - \\bar{\\alpha}_{t-1})})\n= \\color{green}{\\frac{1 - \\bar{\\alpha}_{t-1}}{1 - \\bar{\\alpha}_t} \\cdot \\beta_t} \\\\\n\\tilde{\\boldsymbol{\\mu}}_t (\\mathbf{x}_t, \\mathbf{x}_0)\n&= (\\frac{\\sqrt{\\alpha_t}}{\\beta_t} \\mathbf{x}_t + \\frac{\\sqrt{\\bar{\\alpha}_{t-1} }}{1 - \\bar{\\alpha}_{t-1}} \\mathbf{x}_0)/(\\frac{\\alpha_t}{\\beta_t} + \\frac{1}{1 - \\bar{\\alpha}_{t-1}}) \\\\\n&= (\\frac{\\sqrt{\\alpha_t}}{\\beta_t} \\mathbf{x}_t + \\frac{\\sqrt{\\bar{\\alpha}_{t-1} }}{1 - \\bar{\\alpha}_{t-1}} \\mathbf{x}_0) \\color{green}{\\frac{1 - \\bar{\\alpha}_{t-1}}{1 - \\bar{\\alpha}_t} \\cdot \\beta_t} \\\\\n&= \\frac{\\sqrt{\\alpha_t}(1 - \\bar{\\alpha}_{t-1})}{1 - \\bar{\\alpha}_t} \\mathbf{x}_t + \\frac{\\sqrt{\\bar{\\alpha}_{t-1}}\\beta_t}{1 - \\bar{\\alpha}_t} \\mathbf{x}_0\\\\\n\\end{aligned}\n$$\nThanks to the\nnice property\n, we can represent $\\mathbf{x}_0 = \\frac{1}{\\sqrt{\\bar{\\alpha}_t}}(\\mathbf{x}_t - \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon}_t)$ and plug it into the above equation and obtain:\n$$\n\\begin{aligned}\n\\tilde{\\boldsymbol{\\mu}}_t\n&= \\frac{\\sqrt{\\alpha_t}(1 - \\bar{\\alpha}_{t-1})}{1 - \\bar{\\alpha}_t} \\mathbf{x}_t + \\frac{\\sqrt{\\bar{\\alpha}_{t-1}}\\beta_t}{1 - \\bar{\\alpha}_t} \\frac{1}{\\sqrt{\\bar{\\alpha}_t}}(\\mathbf{x}_t - \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon}_t) \\\\\n&= \\color{cyan}{\\frac{1}{\\sqrt{\\alpha_t}} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_t \\Big)}\n\\end{aligned}\n$$\nAs demonstrated in Fig. 2., such a setup is very similar to\nVAE\nand thus we can use the variational lower bound to optimize the negative log-likelihood.\n$$\n\\begin{aligned}\n- \\log p_\\theta(\\mathbf{x}_0) \n&\\leq - \\log p_\\theta(\\mathbf{x}_0) + D_\\text{KL}(q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0) \\| p_\\theta(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0) ) & \\small{\\text{; KL is non-negative}}\\\\\n&= - \\log p_\\theta(\\mathbf{x}_0) + \\mathbb{E}_{\\mathbf{x}_{1:T}\\sim q(\\mathbf{x}_{1:T} \\vert \\mathbf{x}_0)} \\Big[ \\log\\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T}) / p_\\theta(\\mathbf{x}_0)} \\Big] \\\\\n&= - \\log p_\\theta(\\mathbf{x}_0) + \\mathbb{E}_q \\Big[ \\log\\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T})} + \\log p_\\theta(\\mathbf{x}_0) \\Big] \\\\\n&= \\mathbb{E}_q \\Big[ \\log \\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T})} \\Big] \\\\\n\\text{Let }L_\\text{VLB} \n&= \\mathbb{E}_{q(\\mathbf{x}_{0:T})} \\Big[ \\log \\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T})} \\Big] \\geq - \\mathbb{E}_{q(\\mathbf{x}_0)} \\log p_\\theta(\\mathbf{x}_0)\n\\end{aligned}\n$$\nIt is also straightforward to get the same result using Jensen’s inequality. Say we want to minimize the cross entropy as the learning objective,\n$$\n\\begin{aligned}\nL_\\text{CE}\n&= - \\mathbb{E}_{q(\\mathbf{x}_0)} \\log p_\\theta(\\mathbf{x}_0) \\\\\n&= - \\mathbb{E}_{q(\\mathbf{x}_0)} \\log \\Big( \\int p_\\theta(\\mathbf{x}_{0:T}) d\\mathbf{x}_{1:T} \\Big) \\\\\n&= - \\mathbb{E}_{q(\\mathbf{x}_0)} \\log \\Big( \\int q(\\mathbf{x}_{1:T} \\vert \\mathbf{x}_0) \\frac{p_\\theta(\\mathbf{x}_{0:T})}{q(\\mathbf{x}_{1:T} \\vert \\mathbf{x}_{0})} d\\mathbf{x}_{1:T} \\Big) \\\\\n&= - \\mathbb{E}_{q(\\mathbf{x}_0)} \\log \\Big( \\mathbb{E}_{q(\\mathbf{x}_{1:T} \\vert \\mathbf{x}_0)} \\frac{p_\\theta(\\mathbf{x}_{0:T})}{q(\\mathbf{x}_{1:T} \\vert \\mathbf{x}_{0})} \\Big) \\\\\n&\\leq - \\mathbb{E}_{q(\\mathbf{x}_{0:T})} \\log \\frac{p_\\theta(\\mathbf{x}_{0:T})}{q(\\mathbf{x}_{1:T} \\vert \\mathbf{x}_{0})} \\\\\n&= \\mathbb{E}_{q(\\mathbf{x}_{0:T})}\\Big[\\log \\frac{q(\\mathbf{x}_{1:T} \\vert \\mathbf{x}_{0})}{p_\\theta(\\mathbf{x}_{0:T})} \\Big] = L_\\text{VLB}\n\\end{aligned}\n$$\nTo convert each term in the equation to be analytically computable, the objective can be further rewritten to be a combination of several KL-divergence and entropy terms (See the detailed step-by-step process in Appendix B in\nSohl-Dickstein et al., 2015\n):\n$$\n\\begin{aligned}\nL_\\text{VLB} \n&= \\mathbb{E}_{q(\\mathbf{x}_{0:T})} \\Big[ \\log\\frac{q(\\mathbf{x}_{1:T}\\vert\\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{0:T})} \\Big] \\\\\n&= \\mathbb{E}_q \\Big[ \\log\\frac{\\prod_{t=1}^T q(\\mathbf{x}_t\\vert\\mathbf{x}_{t-1})}{ p_\\theta(\\mathbf{x}_T) \\prod_{t=1}^T p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t) } \\Big] \\\\\n&= \\mathbb{E}_q \\Big[ -\\log p_\\theta(\\mathbf{x}_T) + \\sum_{t=1}^T \\log \\frac{q(\\mathbf{x}_t\\vert\\mathbf{x}_{t-1})}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)} \\Big] \\\\\n&= \\mathbb{E}_q \\Big[ -\\log p_\\theta(\\mathbf{x}_T) + \\sum_{t=2}^T \\log \\frac{q(\\mathbf{x}_t\\vert\\mathbf{x}_{t-1})}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)} + \\log\\frac{q(\\mathbf{x}_1 \\vert \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)} \\Big] \\\\\n&= \\mathbb{E}_q \\Big[ -\\log p_\\theta(\\mathbf{x}_T) + \\sum_{t=2}^T \\log \\Big( \\frac{q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)}\\cdot \\frac{q(\\mathbf{x}_t \\vert \\mathbf{x}_0)}{q(\\mathbf{x}_{t-1}\\vert\\mathbf{x}_0)} \\Big) + \\log \\frac{q(\\mathbf{x}_1 \\vert \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)} \\Big] \\\\\n&= \\mathbb{E}_q \\Big[ -\\log p_\\theta(\\mathbf{x}_T) + \\sum_{t=2}^T \\log \\frac{q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)} + \\sum_{t=2}^T \\log \\frac{q(\\mathbf{x}_t \\vert \\mathbf{x}_0)}{q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_0)} + \\log\\frac{q(\\mathbf{x}_1 \\vert \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)} \\Big] \\\\\n&= \\mathbb{E}_q \\Big[ -\\log p_\\theta(\\mathbf{x}_T) + \\sum_{t=2}^T \\log \\frac{q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)} + \\log\\frac{q(\\mathbf{x}_T \\vert \\mathbf{x}_0)}{q(\\mathbf{x}_1 \\vert \\mathbf{x}_0)} + \\log \\frac{q(\\mathbf{x}_1 \\vert \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)} \\Big]\\\\\n&= \\mathbb{E}_q \\Big[ \\log\\frac{q(\\mathbf{x}_T \\vert \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_T)} + \\sum_{t=2}^T \\log \\frac{q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0)}{p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t)} - \\log p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1) \\Big] \\\\\n&= \\mathbb{E}_q [\\underbrace{D_\\text{KL}(q(\\mathbf{x}_T \\vert \\mathbf{x}_0) \\parallel p_\\theta(\\mathbf{x}_T))}_{L_T} + \\sum_{t=2}^T \\underbrace{D_\\text{KL}(q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0) \\parallel p_\\theta(\\mathbf{x}_{t-1} \\vert\\mathbf{x}_t))}_{L_{t-1}} \\underbrace{- \\log p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)}_{L_0} ]\n\\end{aligned}\n$$\nLet’s label each component in the variational lower bound loss separately:\n$$\n\\begin{aligned}\nL_\\text{VLB} &= L_T + L_{T-1} + \\dots + L_0 \\\\\n\\text{where } L_T &= D_\\text{KL}(q(\\mathbf{x}_T \\vert \\mathbf{x}_0) \\parallel p_\\theta(\\mathbf{x}_T)) \\\\\nL_t &= D_\\text{KL}(q(\\mathbf{x}_t \\vert \\mathbf{x}_{t+1}, \\mathbf{x}_0) \\parallel p_\\theta(\\mathbf{x}_t \\vert\\mathbf{x}_{t+1})) \\text{ for }1 \\leq t \\leq T-1 \\\\\nL_0 &= - \\log p_\\theta(\\mathbf{x}_0 \\vert \\mathbf{x}_1)\n\\end{aligned}\n$$\nEvery KL term in $L_\\text{VLB}$ (except for $L_0$) compares two Gaussian distributions and therefore they can be computed in\nclosed form\n. $L_T$ is constant and can be ignored during training because $q$ has no learnable parameters and $\\mathbf{x}_T$ is a Gaussian noise.\nHo et al. 2020\nmodels $L_0$ using a separate discrete decoder derived from $\\mathcal{N}(\\mathbf{x}_0; \\boldsymbol{\\mu}_\\theta(\\mathbf{x}_1, 1), \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_1, 1))$.\nParameterization of $L_t$ for Training Loss\n#\nRecall that we need to learn a neural network to approximate the conditioned probability distributions in the reverse diffusion process, $p_\\theta(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t) = \\mathcal{N}(\\mathbf{x}_{t-1}; \\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t), \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t))$. We would like to train $\\boldsymbol{\\mu}_\\theta$ to predict $\\tilde{\\boldsymbol{\\mu}}_t = \\frac{1}{\\sqrt{\\alpha_t}} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_t \\Big)$. Because $\\mathbf{x}_t$ is available as input at training time, we can reparameterize the Gaussian noise term instead to make it predict $\\boldsymbol{\\epsilon}_t$ from the input $\\mathbf{x}_t$ at time step $t$:\n$$\n\\begin{aligned}\n\\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t) &= \\color{cyan}{\\frac{1}{\\sqrt{\\alpha_t}} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t) \\Big)} \\\\\n\\text{Thus }\\mathbf{x}_{t-1} &= \\mathcal{N}(\\mathbf{x}_{t-1}; \\frac{1}{\\sqrt{\\alpha_t}} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t) \\Big), \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t))\n\\end{aligned}\n$$\nThe loss term $L_t$ is parameterized to minimize the difference from $\\tilde{\\boldsymbol{\\mu}}$ :\n$$\n\\begin{aligned}\nL_t \n&= \\mathbb{E}_{\\mathbf{x}_0, \\boldsymbol{\\epsilon}} \\Big[\\frac{1}{2 \\| \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t) \\|^2_2} \\| \\color{blue}{\\tilde{\\boldsymbol{\\mu}}_t(\\mathbf{x}_t, \\mathbf{x}_0)} - \\color{green}{\\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t)} \\|^2 \\Big] \\\\\n&= \\mathbb{E}_{\\mathbf{x}_0, \\boldsymbol{\\epsilon}} \\Big[\\frac{1}{2  \\|\\boldsymbol{\\Sigma}_\\theta \\|^2_2} \\| \\color{blue}{\\frac{1}{\\sqrt{\\alpha_t}} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_t \\Big)} - \\color{green}{\\frac{1}{\\sqrt{\\alpha_t}} \\Big( \\mathbf{x}_t - \\frac{1 - \\alpha_t}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\boldsymbol{\\epsilon}}_\\theta(\\mathbf{x}_t, t) \\Big)} \\|^2 \\Big] \\\\\n&= \\mathbb{E}_{\\mathbf{x}_0, \\boldsymbol{\\epsilon}} \\Big[\\frac{ (1 - \\alpha_t)^2 }{2 \\alpha_t (1 - \\bar{\\alpha}_t) \\| \\boldsymbol{\\Sigma}_\\theta \\|^2_2} \\|\\boldsymbol{\\epsilon}_t - \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)\\|^2 \\Big] \\\\\n&= \\mathbb{E}_{\\mathbf{x}_0, \\boldsymbol{\\epsilon}} \\Big[\\frac{ (1 - \\alpha_t)^2 }{2 \\alpha_t (1 - \\bar{\\alpha}_t) \\| \\boldsymbol{\\Sigma}_\\theta \\|^2_2} \\|\\boldsymbol{\\epsilon}_t - \\boldsymbol{\\epsilon}_\\theta(\\sqrt{\\bar{\\alpha}_t}\\mathbf{x}_0 + \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon}_t, t)\\|^2 \\Big] \n\\end{aligned}\n$$\nSimplification\n#\nEmpirically,\nHo et al. (2020)\nfound that training the diffusion model works better with a simplified objective that ignores the weighting term:\n$$\n\\begin{aligned}\nL_t^\\text{simple}\n&= \\mathbb{E}_{t \\sim [1, T], \\mathbf{x}_0, \\boldsymbol{\\epsilon}_t} \\Big[\\|\\boldsymbol{\\epsilon}_t - \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)\\|^2 \\Big] \\\\\n&= \\mathbb{E}_{t \\sim [1, T], \\mathbf{x}_0, \\boldsymbol{\\epsilon}_t} \\Big[\\|\\boldsymbol{\\epsilon}_t - \\boldsymbol{\\epsilon}_\\theta(\\sqrt{\\bar{\\alpha}_t}\\mathbf{x}_0 + \\sqrt{1 - \\bar{\\alpha}_t}\\boldsymbol{\\epsilon}_t, t)\\|^2 \\Big]\n\\end{aligned}\n$$\nThe final simple objective is:\n$$\nL_\\text{simple} = L_t^\\text{simple} + C\n$$\nwhere $C$ is a constant not depending on $\\theta$.\nThe training and sampling algorithms in DDPM (Image source:\nHo et al. 2020\n)\nConnection with noise-conditioned score networks (NCSN)\n#\nSong & Ermon (2019)\nproposed a score-based generative modeling method where samples are produced via\nLangevin dynamics\nusing gradients of the data distribution estimated with score matching. The score of each sample $\\mathbf{x}$’s density probability is defined as its gradient $\\nabla_{\\mathbf{x}} \\log q(\\mathbf{x})$. A score network $\\mathbf{s}_\\theta: \\mathbb{R}^D \\to \\mathbb{R}^D$ is trained to estimate it, $\\mathbf{s}_\\theta(\\mathbf{x}) \\approx \\nabla_{\\mathbf{x}} \\log q(\\mathbf{x})$.\nTo make it scalable with high-dimensional data in the deep learning setting, they proposed to use either\ndenoising score matching\n(\nVincent, 2011\n) or\nsliced score matching\n(use random projections;\nSong et al., 2019\n). Denosing score matching adds a pre-specified small noise to the data $q(\\tilde{\\mathbf{x}} \\vert \\mathbf{x})$ and estimates $q(\\tilde{\\mathbf{x}})$ with score matching.\nRecall that Langevin dynamics can sample data points from a probability density distribution using only the score $\\nabla_{\\mathbf{x}} \\log q(\\mathbf{x})$ in an iterative process.\nHowever, according to the manifold hypothesis, most of the data is expected to concentrate in a low dimensional manifold, even though the observed data might look only arbitrarily high-dimensional. It brings a negative effect on score estimation since the data points cannot cover the whole space. In regions where data density is low, the score estimation is less reliable. After adding a small Gaussian noise to make the perturbed data distribution cover the full space $\\mathbb{R}^D$, the training of the score estimator network becomes more stable.\nSong & Ermon (2019)\nimproved it by perturbing the data with the noise of\ndifferent levels\nand train a noise-conditioned score network to\njointly\nestimate the scores of all the perturbed data at different noise levels.\nThe schedule of increasing noise levels resembles the forward diffusion process. If we use the diffusion process annotation, the score approximates $\\mathbf{s}_\\theta(\\mathbf{x}_t, t) \\approx \\nabla_{\\mathbf{x}_t} \\log q(\\mathbf{x}_t)$. Given a Gaussian distribution $\\mathbf{x} \\sim \\mathcal{N}(\\mathbf{\\mu}, \\sigma^2 \\mathbf{I})$, we can write the derivative of the logarithm of its density function as $\\nabla_{\\mathbf{x}}\\log p(\\mathbf{x}) = \\nabla_{\\mathbf{x}} \\Big(-\\frac{1}{2\\sigma^2}(\\mathbf{x} - \\boldsymbol{\\mu})^2 \\Big) = - \\frac{\\mathbf{x} - \\boldsymbol{\\mu}}{\\sigma^2} = - \\frac{\\boldsymbol{\\epsilon}}{\\sigma}$ where $\\boldsymbol{\\epsilon} \\sim \\mathcal{N}(\\boldsymbol{0}, \\mathbf{I})$.\nRecall\nthat $q(\\mathbf{x}_t \\vert \\mathbf{x}_0) \\sim \\mathcal{N}(\\sqrt{\\bar{\\alpha}_t} \\mathbf{x}_0, (1 - \\bar{\\alpha}_t)\\mathbf{I})$ and therefore,\n$$\n\\mathbf{s}_\\theta(\\mathbf{x}_t, t) \n\\approx \\nabla_{\\mathbf{x}_t} \\log q(\\mathbf{x}_t)\n= \\mathbb{E}_{q(\\mathbf{x}_0)} [\\nabla_{\\mathbf{x}_t} \\log q(\\mathbf{x}_t \\vert \\mathbf{x}_0)]\n= \\mathbb{E}_{q(\\mathbf{x}_0)} \\Big[ - \\frac{\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)}{\\sqrt{1 - \\bar{\\alpha}_t}} \\Big]\n= - \\frac{\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)}{\\sqrt{1 - \\bar{\\alpha}_t}}\n$$\nParameterization of $\\beta_t$\n#\nThe forward variances are set to be a sequence of linearly increasing constants in\nHo et al. (2020)\n, from $\\beta_1=10^{-4}$ to $\\beta_T=0.02$. They are relatively small compared to the normalized image pixel values between $[-1, 1]$. Diffusion models in their experiments showed high-quality samples but still could not achieve competitive model log-likelihood as other generative models.\nNichol & Dhariwal (2021)\nproposed several improvement techniques to help diffusion models to obtain lower NLL. One of the improvements is to use a cosine-based variance schedule. The choice of the scheduling function can be arbitrary, as long as it provides a near-linear drop in the middle of the training process and subtle changes around $t=0$ and $t=T$.\n$$\n\\beta_t = \\text{clip}(1-\\frac{\\bar{\\alpha}_t}{\\bar{\\alpha}_{t-1}}, 0.999) \\quad\\bar{\\alpha}_t = \\frac{f(t)}{f(0)}\\quad\\text{where }f(t)=\\cos\\Big(\\frac{t/T+s}{1+s}\\cdot\\frac{\\pi}{2}\\Big)^2\n$$\nwhere the small offset $s$ is to prevent $\\beta_t$ from being too small when close to $t=0$.\nComparison of linear and cosine-based scheduling of $\\beta\\_t$ during training. (Image source:\nNichol & Dhariwal, 2021\n)\nParameterization of reverse process variance $\\boldsymbol{\\Sigma}_\\theta$\n#\nHo et al. (2020)\nchose to fix $\\beta_t$ as constants instead of making them learnable and set $\\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t) = \\sigma^2_t \\mathbf{I}$ , where $\\sigma_t$ is not learned but set to $\\beta_t$ or $\\tilde{\\beta}_t = \\frac{1 - \\bar{\\alpha}_{t-1}}{1 - \\bar{\\alpha}_t} \\cdot \\beta_t$. Because they found that learning a diagonal variance $\\boldsymbol{\\Sigma}_\\theta$ leads to unstable training and poorer sample quality.\nNichol & Dhariwal (2021)\nproposed to learn $\\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t)$ as an interpolation between $\\beta_t$ and $\\tilde{\\beta}_t$ by model predicting a mixing vector $\\mathbf{v}$ :\n$$\n\\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t) = \\exp(\\mathbf{v} \\log \\beta_t + (1-\\mathbf{v}) \\log \\tilde{\\beta}_t)\n$$\nHowever, the simple objective $L_\\text{simple}$ does not depend on $\\boldsymbol{\\Sigma}_\\theta$ . To add the dependency, they constructed a hybrid objective $L_\\text{hybrid} = L_\\text{simple} + \\lambda L_\\text{VLB}$ where $\\lambda=0.001$ is small and stop gradient on $\\boldsymbol{\\mu}_\\theta$ in the $L_\\text{VLB}$ term such that $L_\\text{VLB}$ only guides the learning of $\\boldsymbol{\\Sigma}_\\theta$. Empirically they observed that $L_\\text{VLB}$ is pretty challenging to optimize likely due to noisy gradients, so they proposed to use a time-averaging smoothed version of $L_\\text{VLB}$ with importance sampling.\nComparison of negative log-likelihood of improved DDPM with other likelihood-based generative models. NLL is reported in the unit of bits/dim. (Image source:\nNichol & Dhariwal, 2021\n)\nConditioned Generation\n#\nWhile training generative models on images with conditioning information such as ImageNet dataset, it is common to generate samples conditioned on class labels or a piece of descriptive text.\nClassifier Guided Diffusion\n#\nTo explicit incorporate class information into the diffusion process,\nDhariwal & Nichol (2021)\ntrained a classifier $f_\\phi(y \\vert \\mathbf{x}_t, t)$ on noisy image $\\mathbf{x}_t$ and use gradients $\\nabla_\\mathbf{x} \\log f_\\phi(y \\vert \\mathbf{x}_t)$ to guide the diffusion sampling process toward the conditioning information $y$ (e.g. a target class label) by altering the noise prediction.\nRecall\nthat $\\nabla_{\\mathbf{x}_t} \\log q(\\mathbf{x}_t) = - \\frac{1}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)$ and we can write the score function for the joint distribution $q(\\mathbf{x}_t, y)$ as following,\n$$\n\\begin{aligned}\n\\nabla_{\\mathbf{x}_t} \\log q(\\mathbf{x}_t, y)\n&= \\nabla_{\\mathbf{x}_t} \\log q(\\mathbf{x}_t) + \\nabla_{\\mathbf{x}_t} \\log q(y \\vert \\mathbf{x}_t) \\\\\n&\\approx - \\frac{1}{\\sqrt{1 - \\bar{\\alpha}_t}} \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t) + \\nabla_{\\mathbf{x}_t} \\log f_\\phi(y \\vert \\mathbf{x}_t) \\\\\n&= - \\frac{1}{\\sqrt{1 - \\bar{\\alpha}_t}} (\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t) - \\sqrt{1 - \\bar{\\alpha}_t} \\nabla_{\\mathbf{x}_t} \\log f_\\phi(y \\vert \\mathbf{x}_t))\n\\end{aligned}\n$$\nThus, a new classifier-guided predictor $\\bar{\\boldsymbol{\\epsilon}}_\\theta$ would take the form as following,\n$$\n\\bar{\\boldsymbol{\\epsilon}}_\\theta(\\mathbf{x}_t, t) = \\boldsymbol{\\epsilon}_\\theta(x_t, t) - \\sqrt{1 - \\bar{\\alpha}_t} \\nabla_{\\mathbf{x}_t} \\log f_\\phi(y \\vert \\mathbf{x}_t)\n$$\nTo control the strength of the classifier guidance, we can add a weight $w$ to the delta part,\n$$\n\\bar{\\boldsymbol{\\epsilon}}_\\theta(\\mathbf{x}_t, t) = \\boldsymbol{\\epsilon}_\\theta(x_t, t) - \\sqrt{1 - \\bar{\\alpha}_t} \\; w \\nabla_{\\mathbf{x}_t} \\log f_\\phi(y \\vert \\mathbf{x}_t)\n$$\nThe resulting\nablated diffusion model\n(\nADM\n) and the one with additional classifier guidance (\nADM-G\n) are able to achieve better results than SOTA generative models (e.g. BigGAN).\nThe algorithms use guidance from a classifier to run conditioned generation with DDPM and DDIM. (Image source:\nDhariwal & Nichol, 2021\n])\nAdditionally with some modifications on the U-Net architecture,\nDhariwal & Nichol (2021)\nshowed performance better than GAN with diffusion models. The architecture modifications include larger model depth/width, more attention heads, multi-resolution attention, BigGAN residual blocks for up/downsampling, residual connection rescale by $1/\\sqrt{2}$ and adaptive group normalization (AdaGN).\nClassifier-Free Guidance\n#\nWithout an independent classifier $f_\\phi$, it is still possible to run conditional diffusion steps by incorporating the scores from a conditional and an unconditional diffusion model (\nHo & Salimans, 2021\n). Let unconditional denoising diffusion model $p_\\theta(\\mathbf{x})$ parameterized through a score estimator $\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)$ and the conditional model $p_\\theta(\\mathbf{x} \\vert y)$ parameterized through $\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t, y)$. These two models can be learned via a single neural network. Precisely, a conditional diffusion model $p_\\theta(\\mathbf{x} \\vert y)$ is trained on paired data $(\\mathbf{x}, y)$, where the conditioning information $y$ gets discarded periodically at random such that the model knows how to generate images unconditionally as well, i.e. $\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t) = \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t, y=\\varnothing)$.\nThe gradient of an implicit classifier can be represented with conditional and unconditional score estimators. Once plugged into the classifier-guided modified score, the score contains no dependency on a separate classifier.\n$$\n\\begin{aligned}\n\\nabla_{\\mathbf{x}_t} \\log p(y \\vert \\mathbf{x}_t)\n&= \\nabla_{\\mathbf{x}_t} \\log p(\\mathbf{x}_t \\vert y) - \\nabla_{\\mathbf{x}_t} \\log p(\\mathbf{x}_t) \\\\\n&= - \\frac{1}{\\sqrt{1 - \\bar{\\alpha}_t}}\\Big( \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t, y) - \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t) \\Big) \\\\\n\\bar{\\boldsymbol{\\epsilon}}_\\theta(\\mathbf{x}_t, t, y)\n&= \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t, y) - \\sqrt{1 - \\bar{\\alpha}_t} \\; w \\nabla_{\\mathbf{x}_t} \\log p(y \\vert \\mathbf{x}_t) \\\\\n&= \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t, y) + w \\big(\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t, y) - \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t) \\big) \\\\\n&= (w+1) \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t, y) - w \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)\n\\end{aligned}\n$$\nTheir experiments showed that classifier-free guidance can achieve a good balance between FID (distinguish between synthetic and generated images) and IS (quality and diversity).\nThe guided diffusion model, GLIDE (\nNichol, Dhariwal & Ramesh, et al. 2022\n), explored both guiding strategies, CLIP guidance and classifier-free guidance, and found that the latter is more preferred. They hypothesized that it is because CLIP guidance exploits the model with adversarial examples towards the CLIP model, rather than optimize the better matched images generation.\nSpeed up Diffusion Models\n#\nIt is very slow to generate a sample from DDPM by following the Markov chain of the reverse diffusion process, as $T$ can be up to one or a few thousand steps. One data point from\nSong et al. (2020)\n: “For example, it takes around 20 hours to sample 50k images of size 32 × 32 from a DDPM, but less than a minute to do so from a GAN on an Nvidia 2080 Ti GPU.”\nFewer Sampling Steps & Distillation\n#\nOne simple way is to run a strided sampling schedule (\nNichol & Dhariwal, 2021\n) by taking the sampling update every $\\lceil T/S \\rceil$ steps to reduce the process from $T$ to $S$ steps. The new sampling schedule for generation is $\\{\\tau_1, \\dots, \\tau_S\\}$  where $\\tau_1 < \\tau_2 < \\dots <\\tau_S \\in [1, T]$ and $S < T$.\nFor another approach, let’s rewrite $q_\\sigma(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0)$ to be parameterized by a desired standard deviation $\\sigma_t$ according to the\nnice property\n:\n$$\n\\begin{aligned}\n\\mathbf{x}_{t-1} \n&= \\sqrt{\\bar{\\alpha}_{t-1}}\\mathbf{x}_0 +  \\sqrt{1 - \\bar{\\alpha}_{t-1}}\\boldsymbol{\\epsilon}_{t-1} & \\\\\n&= \\sqrt{\\bar{\\alpha}_{t-1}}\\mathbf{x}_0 + \\sqrt{1 - \\bar{\\alpha}_{t-1} - \\sigma_t^2} \\boldsymbol{\\epsilon}_t + \\sigma_t\\boldsymbol{\\epsilon} & \\\\\n&= \\sqrt{\\bar{\\alpha}_{t-1}} \\Big( \\frac{\\mathbf{x}_t - \\sqrt{1 - \\bar{\\alpha}_t} \\epsilon^{(t)}_\\theta(\\mathbf{x}_t)}{\\sqrt{\\bar{\\alpha}_t}} \\Big) + \\sqrt{1 - \\bar{\\alpha}_{t-1} - \\sigma_t^2} \\epsilon^{(t)}_\\theta(\\mathbf{x}_t) + \\sigma_t\\boldsymbol{\\epsilon} \\\\\nq_\\sigma(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0)\n&= \\mathcal{N}(\\mathbf{x}_{t-1}; \\sqrt{\\bar{\\alpha}_{t-1}} \\Big( \\frac{\\mathbf{x}_t - \\sqrt{1 - \\bar{\\alpha}_t} \\epsilon^{(t)}_\\theta(\\mathbf{x}_t)}{\\sqrt{\\bar{\\alpha}_t}} \\Big) + \\sqrt{1 - \\bar{\\alpha}_{t-1} - \\sigma_t^2} \\epsilon^{(t)}_\\theta(\\mathbf{x}_t), \\sigma_t^2 \\mathbf{I})\n\\end{aligned}\n$$\nwhere the model $\\epsilon^{(t)}_\\theta(.)$ predicts the $\\epsilon_t$ from $\\mathbf{x}_t$.\nRecall that in $q(\\mathbf{x}_{t-1} \\vert \\mathbf{x}_t, \\mathbf{x}_0) = \\mathcal{N}(\\mathbf{x}_{t-1}; \\tilde{\\boldsymbol{\\mu}}(\\mathbf{x}_t, \\mathbf{x}_0), \\tilde{\\beta}_t \\mathbf{I})$, therefore we have:\n$$\n\\tilde{\\beta}_t = \\sigma_t^2 = \\frac{1 - \\bar{\\alpha}_{t-1}}{1 - \\bar{\\alpha}_t} \\cdot \\beta_t\n$$\nLet $\\sigma_t^2 = \\eta \\cdot \\tilde{\\beta}_t$ such that we can adjust $\\eta \\in \\mathbb{R}^+$ as a hyperparameter to control the sampling stochasticity. The special case of $\\eta = 0$ makes the sampling process\ndeterministic\n. Such a model is named the\ndenoising diffusion implicit model\n(\nDDIM\n;\nSong et al., 2020\n). DDIM has the same marginal noise distribution but deterministically maps noise back to the original data samples.\nDuring generation, we don’t have to follow the whole chain $t=1,\\dots,T$, but rather a subset of steps. Let’s denote $s < t$ as two steps in this accelerated trajectory. The DDIM update step is:\n$$\nq_{\\sigma, s < t}(\\mathbf{x}_s \\vert \\mathbf{x}_t, \\mathbf{x}_0)\n= \\mathcal{N}(\\mathbf{x}_s; \\sqrt{\\bar{\\alpha}_s} \\Big( \\frac{\\mathbf{x}_t - \\sqrt{1 - \\bar{\\alpha}_t} \\epsilon^{(t)}_\\theta(\\mathbf{x}_t)}{\\sqrt{\\bar{\\alpha}_t}} \\Big) + \\sqrt{1 - \\bar{\\alpha}_s - \\sigma_t^2} \\epsilon^{(t)}_\\theta(\\mathbf{x}_t), \\sigma_t^2 \\mathbf{I})\n$$\nWhile all the models are trained with $T=1000$ diffusion steps in the experiments, they observed that DDIM ($\\eta=0$) can produce the best quality samples when $S$ is small, while DDPM ($\\eta=1$) performs much worse on small $S$. DDPM does perform better when we can afford to run the full reverse Markov diffusion steps ($S=T=1000$). With DDIM, it is possible to train the diffusion model up to any arbitrary number of forward steps but only sample from a subset of steps in the generative process.\nFID scores on CIFAR10 and CelebA datasets by diffusion models of different settings, including $\\color{cyan}{\\text{DDIM}}$ ($\\eta=0$) and $\\color{orange}{\\text{DDPM}}$ ($\\hat{\\sigma}$). (Image source:\nSong et al., 2020\n)\nCompared to DDPM, DDIM is able to:\nGenerate higher-quality samples using a much fewer number of steps.\nHave “consistency” property since the generative process is deterministic, meaning that multiple samples conditioned on the same latent variable should have similar high-level features.\nBecause of the consistency, DDIM can do semantically meaningful interpolation in the latent variable.\nProgressive distillation can reduce the diffusion sampling steps by half in each iteration. (Image source:\nSalimans & Ho, 2022\n)\nProgressive Distillation\n(\nSalimans & Ho, 2022\n) is a method for distilling trained deterministic samplers into new models of halved sampling steps. The student model is initialized from the teacher model and denoises towards a target where one student DDIM step matches 2 teacher steps, instead of using the original sample $\\mathbf{x}_0$ as the denoise target. In every progressive distillation iteration, we can half the sampling steps.\nComparison of Algorithm 1 (diffusion model training) and Algorithm 2 (progressive distillation) side-by-side, where the relative changes in progressive distillation are highlighted in green.\n(Image source:\nSalimans & Ho, 2022\n)\nConsistency Models\n(\nSong et al. 2023\n) learns to map any intermediate noisy data points $\\mathbf{x}_t, t > 0$ on the diffusion sampling trajectory back to its origin $\\mathbf{x}_0$ directly. It is named as\nconsistency\nmodel because of its\nself-consistency\nproperty as any data points on the same trajectory is mapped to the same origin.\nConsistency models learn to map any data point on the trajectory back to its origin. (Image source:\nSong et al., 2023\n)\nGiven a trajectory $\\{\\mathbf{x}_t \\vert t \\in [\\epsilon, T]\\}$ , the\nconsistency function\n$f$ is defined as $f: (\\mathbf{x}_t, t) \\mapsto \\mathbf{x}_\\epsilon$ and the equation $f(\\mathbf{x}_t, t) = f(\\mathbf{x}_{t’}, t’) = \\mathbf{x}_\\epsilon$ holds true for all $t, t’ \\in [\\epsilon, T]$. When $t=\\epsilon$, $f$ is an identify function. The model can be parameterized as follows, where $c_\\text{skip}(t)$ and $c_\\text{out}(t)$ functions are designed in a way that $c_\\text{skip}(\\epsilon) = 1, c_\\text{out}(\\epsilon) = 0$:\n$$\nf_\\theta(\\mathbf{x}, t) = c_\\text{skip}(t)\\mathbf{x} + c_\\text{out}(t) F_\\theta(\\mathbf{x}, t)\n$$\nIt is possible for the consistency model to generate samples in a single step, while still maintaining the flexibility of trading computation for better quality following a multi-step sampling process.\nThe paper introduced two ways to train consistency models:\nConsistency Distillation (CD)\n: Distill a diffusion model into a consistency model by minimizing the difference between model outputs for pairs generated out of the same trajectory. This enables a much cheaper sampling evaluation. The consistency distillation loss is:\n$$\n \\begin{aligned}\n \\mathcal{L}^N_\\text{CD} (\\theta, \\theta^-; \\phi) &= \\mathbb{E}\n [\\lambda(t_n)d(f_\\theta(\\mathbf{x}_{t_{n+1}}, t_{n+1}), f_{\\theta^-}(\\hat{\\mathbf{x}}^\\phi_{t_n}, t_n)] \\\\\n \\hat{\\mathbf{x}}^\\phi_{t_n} &= \\mathbf{x}_{t_{n+1}} - (t_n - t_{n+1}) \\Phi(\\mathbf{x}_{t_{n+1}}, t_{n+1}; \\phi)\n \\end{aligned}\n $$\nwhere\n$\\Phi(.;\\phi)$ is the update function of a one-step\nODE\nsolver;\n$n \\sim \\mathcal{U}[1, N-1]$, has an uniform distribution over $1, \\dots, N-1$;\nThe network parameters $\\theta^-$ is EMA version of $\\theta$ which greatly stabilizes the training (just like in\nDQN\nor\nmomentum\ncontrastive learning);\n$d(.,.)$ is a positive distance metric function that satisfies $\\forall \\mathbf{x}, \\mathbf{y}: d(\\mathbf{x}, \\mathbf{y}) \\geq 0$ and $d(\\mathbf{x}, \\mathbf{y}) = 0$ if and only if $\\mathbf{x} = \\mathbf{y}$ such as $\\ell_2$, $\\ell_1$ or\nLPIPS\n(learned perceptual image patch similarity) distance;\n$\\lambda(.) \\in \\mathbb{R}^+$ is a positive weighting function and the paper sets $\\lambda(t_n)=1$.\nConsistency Training (CT)\n: The other option is to train a consistency model independently. Note that in CD, a pre-trained score model $s_\\phi(\\mathbf{x}, t)$ is used to approximate the ground truth score $\\nabla\\log p_t(\\mathbf{x})$ but in CT we need a way to estimate this score function and it turns out an unbiased estimator of $\\nabla\\log p_t(\\mathbf{x})$ exists as $-\\frac{\\mathbf{x}_t - \\mathbf{x}}{t^2}$. The CT loss is defined as follows:\n$$\n\\mathcal{L}^N_\\text{CT} (\\theta, \\theta^-; \\phi) = \\mathbb{E}\n[\\lambda(t_n)d(f_\\theta(\\mathbf{x} + t_{n+1} \\mathbf{z},\\;t_{n+1}), f_{\\theta^-}(\\mathbf{x} + t_n \\mathbf{z},\\;t_n)]\n\\text{ where }\\mathbf{z} \\in \\mathcal{N}(\\mathbf{0}, \\mathbf{I})\n$$\nAccording to the experiments in the paper, they found,\nHeun ODE solver works better than Euler’s first-order solver, since higher order ODE solvers have smaller estimation errors with the same $N$.\nAmong different options of the distance metric function $d(.)$, the LPIPS metric works better than $\\ell_1$ and $\\ell_2$ distance.\nSmaller $N$ leads to faster convergence but worse samples, whereas larger $N$ leads to slower convergence but better samples upon convergence.\nComparison of consistency models' performance under different configurations. The best configuration for CD is LPIPS distance metric, Heun ODE solver, and $N=18$.  (Image source:\nSong et al., 2023\n)\nLatent Variable Space\n#\nLatent diffusion model\n(\nLDM\n;\nRombach & Blattmann, et al. 2022\n) runs the diffusion process in the latent space instead of pixel space, making training cost lower and inference speed faster. It is motivated by the observation that most bits of an image contribute to perceptual details and the semantic and conceptual composition still remains after aggressive compression. LDM loosely decomposes the perceptual compression and semantic compression with generative modeling learning by first trimming off pixel-level redundancy with autoencoder and then manipulating / generating semantic concepts with diffusion process on learned latent.\nThe plot for tradeoff between compression rate and distortion, illustrating two-stage compressions - perceptual and semantic compression. (Image source:\nRombach & Blattmann, et al. 2022\n)\nThe perceptual compression process relies on an autoencoder model. An encoder $\\mathcal{E}$ is used to compress the input image $\\mathbf{x} \\in \\mathbb{R}^{H \\times W \\times 3}$ to a smaller 2D latent vector $\\mathbf{z} = \\mathcal{E}(\\mathbf{x}) \\in \\mathbb{R}^{h \\times w \\times c}$ , where the downsampling rate $f=H/h=W/w=2^m, m \\in \\mathbb{N}$. Then an decoder $\\mathcal{D}$ reconstructs the images from the latent vector, $\\tilde{\\mathbf{x}} = \\mathcal{D}(\\mathbf{z})$. The paper explored two types of regularization in autoencoder training to avoid arbitrarily high-variance in the latent spaces.\nKL-reg\n: A small KL penalty towards a standard normal distribution over the learned latent, similar to\nVAE\n.\nVQ-reg\n: Uses a vector quantization layer within the decoder, like\nVQVAE\nbut the quantization layer is absorbed by the decoder.\nThe diffusion and denoising processes happen on the latent vector $\\mathbf{z}$. The denoising model is a time-conditioned U-Net, augmented with the cross-attention mechanism to handle flexible conditioning information for image generation (e.g. class labels, semantic maps, blurred variants of an image). The design is equivalent to fuse representation of different modality into the model with a cross-attention mechanism. Each type of conditioning information is paired with a domain-specific encoder $\\tau_\\theta$ to project the conditioning input $y$ to an intermediate representation that can be mapped into cross-attention component, $\\tau_\\theta(y) \\in \\mathbb{R}^{M \\times d_\\tau}$:\n$$\n\\begin{aligned}\n&\\text{Attention}(\\mathbf{Q}, \\mathbf{K}, \\mathbf{V}) = \\text{softmax}\\Big(\\frac{\\mathbf{Q}\\mathbf{K}^\\top}{\\sqrt{d}}\\Big) \\cdot \\mathbf{V} \\\\\n&\\text{where }\\mathbf{Q} = \\mathbf{W}^{(i)}_Q \\cdot \\varphi_i(\\mathbf{z}_i),\\;\n\\mathbf{K} = \\mathbf{W}^{(i)}_K \\cdot \\tau_\\theta(y),\\;\n\\mathbf{V} = \\mathbf{W}^{(i)}_V \\cdot \\tau_\\theta(y) \\\\\n&\\text{and }\n\\mathbf{W}^{(i)}_Q \\in \\mathbb{R}^{d \\times d^i_\\epsilon},\\;\n\\mathbf{W}^{(i)}_K, \\mathbf{W}^{(i)}_V \\in \\mathbb{R}^{d \\times d_\\tau},\\;\n\\varphi_i(\\mathbf{z}_i) \\in \\mathbb{R}^{N \\times d^i_\\epsilon},\\;\n\\tau_\\theta(y) \\in \\mathbb{R}^{M \\times d_\\tau}\n\\end{aligned}\n$$\nThe architecture of the latent diffusion model (LDM). (Image source:\nRombach & Blattmann, et al. 2022\n)\nScale up Generation Resolution and Quality\n#\nTo generate high-quality images at high resolution,\nHo et al. (2021)\nproposed to use a pipeline of multiple diffusion models at increasing resolutions.\nNoise conditioning augmentation\nbetween pipeline models is crucial to the final image quality, which is to apply strong data augmentation to the conditioning input $\\mathbf{z}$ of each super-resolution model $p_\\theta(\\mathbf{x} \\vert \\mathbf{z})$. The conditioning noise helps reduce compounding error in the pipeline setup.\nU-net\nis a common choice of model architecture in diffusion modeling for high-resolution image generation.\nA cascaded pipeline of multiple diffusion models at increasing resolutions. (Image source:\nHo et al. 2021\n])\nThey found the most effective noise is to apply Gaussian noise at low resolution and Gaussian blur at high resolution. In addition, they also explored two forms of conditioning augmentation that require small modification to the training process. Note that conditioning noise is only applied to training but not at inference.\nTruncated conditioning augmentation stops the diffusion process early at step $t > 0$ for low resolution.\nNon-truncated conditioning augmentation runs the full low resolution reverse process until step 0 but then corrupt it by $\\mathbf{z}_t \\sim q(\\mathbf{x}_t \\vert \\mathbf{x}_0)$ and then feeds the corrupted $\\mathbf{z}_t$ s into the super-resolution model.\nThe two-stage diffusion model\nunCLIP\n(\nRamesh et al. 2022\n) heavily utilizes the CLIP text encoder to produce text-guided images at high quality. Given a pretrained CLIP model $\\mathbf{c}$ and paired training data for the diffusion model, $(\\mathbf{x}, y)$, where $x$ is an image and $y$ is the corresponding caption, we can compute the CLIP text and image embedding, $\\mathbf{c}^t(y)$ and $\\mathbf{c}^i(\\mathbf{x})$, respectively. The unCLIP learns two models in parallel:\nA prior model $P(\\mathbf{c}^i \\vert y)$: outputs CLIP image embedding $\\mathbf{c}^i$ given the text $y$.\nA decoder $P(\\mathbf{x} \\vert \\mathbf{c}^i, [y])$: generates the image $\\mathbf{x}$ given CLIP image embedding $\\mathbf{c}^i$ and optionally the original text $y$.\nThese two models enable conditional generation, because\n$$\n\\underbrace{P(\\mathbf{x} \\vert y) = P(\\mathbf{x}, \\mathbf{c}^i \\vert y)}_{\\mathbf{c}^i\\text{ is deterministic given }\\mathbf{x}} = P(\\mathbf{x} \\vert \\mathbf{c}^i, y)P(\\mathbf{c}^i \\vert y)\n$$\nThe architecture of unCLIP. (Image source:\nRamesh et al. 2022\n])\nunCLIP follows a two-stage image generation process:\nGiven a text $y$, a CLIP model is first used to generate a text embedding $\\mathbf{c}^t(y)$. Using CLIP latent space enables zero-shot image manipulation via text.\nA diffusion or autoregressive prior $P(\\mathbf{c}^i \\vert y)$ processes this CLIP text embedding to construct an image prior and then a diffusion decoder $P(\\mathbf{x} \\vert \\mathbf{c}^i, [y])$ generates an image, conditioned on the prior. This decoder can also generate image variations conditioned on an image input, preserving its style and semantics.\nInstead of CLIP model,\nImagen\n(\nSaharia et al. 2022\n) uses a pre-trained large LM (i.e. a frozen T5-XXL text encoder) to encode text for image generation. There is a general trend that larger model size can lead to better image quality and text-image alignment. They found that T5-XXL and CLIP text encoder achieve similar performance on MS-COCO, but human evaluation prefers T5-XXL on DrawBench (a collection of prompts covering 11 categories).\nWhen applying classifier-free guidance, increasing $w$ may lead to better image-text alignment but worse image fidelity. They found that it is due to train-test mismatch, that is to say, because training data $\\mathbf{x}$ stays within the range $[-1, 1]$, the test data should be so too. Two thresholding strategies are introduced:\nStatic thresholding: clip $\\mathbf{x}$ prediction to $[-1, 1]$\nDynamic thresholding: at each sampling step, compute $s$ as a certain percentile absolute pixel value; if $s > 1$, clip the prediction to $[-s, s]$ and divide by $s$.\nImagen modifies several designs in U-net to make it\nefficient U-Net\n.\nShift model parameters from high resolution blocks to low resolution by adding more residual locks for the lower resolutions;\nScale the skip connections by $1/\\sqrt{2}$\nReverse the order of downsampling (move it before convolutions) and upsampling operations (move it after convolution) in order to improve the speed of forward pass.\nThey found that noise conditioning augmentation, dynamic thresholding and efficient U-Net are critical for image quality, but scaling text encoder size is more important than U-Net size.\nModel Architecture\n#\nThere are two common backbone architecture choices for diffusion models: U-Net and Transformer.\nU-Net\n(\nRonneberger, et al. 2015\n) consists of a downsampling stack and an upsampling stack.\nDownsampling\n: Each step consists of the repeated application of two 3x3 convolutions (unpadded convolutions), each followed by a ReLU and a 2x2 max pooling with stride 2. At each downsampling step, the number of feature channels is doubled.\nUpsampling\n: Each step consists of an upsampling of the feature map followed by a 2x2 convolution and each halves the number of feature channels.\nShortcuts\n: Shortcut connections result in a concatenation with the corresponding layers of the downsampling stack and provide the essential high-resolution features to the upsampling process.\nThe U-net architecture. Each blue square is a feature map with the number of channels labeled on top and the height x width dimension labeled on the left bottom side. The gray arrows mark the shortcut connections. (Image source:\nRonneberger, 2015\n)\nTo enable image generation conditioned on additional images for composition info like Canny edges, Hough lines, user scribbles, human post skeletons, segmentation maps, depths and normals,\nControlNet\n(\nZhang et al. 2023\nintroduces architectural changes via adding a “sandwiched” zero convolution layers of a trainable copy of the original model weights into each encoder layer of the U-Net. Precisely, given a neural network block $\\mathcal{F}_\\theta(.)$, ControlNet does the following:\nFirst, freeze the original parameters $\\theta$ of the original block\nClone it to be a copy with trainable parameters $\\theta_c$  and an additional conditioning vector $\\mathbf{c}$.\nUse two zero convolution layers, denoted as $\\mathcal{Z}_{\\theta_{z1}}(.;.)$ and $\\mathcal{Z}_{\\theta_{z2}}(.;.)$, which is 1x1 convo layers with both weights and biases initialized to be zeros, to connect these two blocks. Zero convolutions protect this back-bone by eliminating random noise as gradients in the initial training steps.\nThe final output is: $\\mathbf{y}_c = \\mathcal{F}_\\theta(\\mathbf{x}) + \\mathcal{Z}_{\\theta_{z2}}(\\mathcal{F}_{\\theta_c}(\\mathbf{x} + \\mathcal{Z}_{\\theta_{z1}}(\\mathbf{c})))$\nThe ControlNet architecture. (Image source:\nZhang et al. 2023\n)\nDiffusion Transformer\n(\nDiT\n;\nPeebles & Xie, 2023\n) for diffusion modeling operates on latent patches, using the same design space of\nLDM\n(Latent Diffusion Model)]. DiT has the following setup:\nTake the latent representation of an input $\\mathbf{z}$ as input to DiT.\n“Patchify” the noise latent of size $I \\times I \\times C$ into patches of size $p$ and convert it into a sequence of patches of size $(I/p)^2$.\nThen this sequence of tokens go through Transformer blocks. They are exploring three different designs for how to do generation conditioned on contextual information like timestep $t$ or class label $c$. Among three designs,\nadaLN (Adaptive layer norm)-Zero\nworks out the best, better than in-context conditioning and cross-attention block. The scale and shift parameters, $\\gamma$ and $\\beta$, are regressed from the sum of the embedding vectors of $t$ and $c$. The dimension-wise scaling parameters $\\alpha$ is also regressed and applied immediately prior to any residual connections within the DiT block.\nThe transformer decoder outputs noise predictions and an output diagonal covariance prediction.\nThe Diffusion Transformer (DiT) architecture.\n(Image source:\nPeebles & Xie, 2023\n)\nTransformer architecture can be easily scaled up and it is well known for that. This is one of the biggest benefits of DiT as its performance scales up with more compute and larger DiT models are more compute efficient according to the experiments.\nQuick Summary\n#\nPros\n: Tractability and flexibility are two conflicting objectives in generative modeling. Tractable models can be analytically evaluated and cheaply fit data (e.g. via a Gaussian or Laplace), but they cannot easily describe the structure in rich datasets. Flexible models can fit arbitrary structures in data, but evaluating, training, or sampling from these models is usually expensive. Diffusion models are both analytically tractable and flexible\nCons\n: Diffusion models rely on a long Markov chain of diffusion steps to generate samples, so it can be quite expensive in terms of time and compute. New methods have been proposed to make the process much faster, but the sampling is still slower than GAN.\nCitation\n#\nCited as:\nWeng, Lilian. (Jul 2021). What are diffusion models? Lil’Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.\nOr\n@article{weng2021diffusion,\n  title   = \"What are diffusion models?\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2021\",\n  month   = \"Jul\",\n  url     = \"https://lilianweng.github.io/posts/2021-07-11-diffusion-models/\"\n}\nReferences\n#\n[1] Jascha Sohl-Dickstein et al.\n“Deep Unsupervised Learning using Nonequilibrium Thermodynamics.”\nICML 2015.\n[2] Max Welling & Yee Whye Teh.\n“Bayesian learning via stochastic gradient langevin dynamics.”\nICML 2011.\n[3] Yang Song & Stefano Ermon.\n“Generative modeling by estimating gradients of the data distribution.”\nNeurIPS 2019.\n[4] Yang Song & Stefano Ermon.\n“Improved techniques for training score-based generative models.”\nNeuriPS 2020.\n[5] Jonathan Ho et al.\n“Denoising diffusion probabilistic models.”\narxiv Preprint arxiv:2006.11239 (2020). [\ncode\n]\n[6] Jiaming Song et al.\n“Denoising diffusion implicit models.”\narxiv Preprint arxiv:2010.02502 (2020). [\ncode\n]\n[7] Alex Nichol & Prafulla Dhariwal.\n“Improved denoising diffusion probabilistic models”\narxiv Preprint arxiv:2102.09672 (2021). [\ncode\n]\n[8] Prafula Dhariwal & Alex Nichol.\n“Diffusion Models Beat GANs on Image Synthesis.”\narxiv Preprint arxiv:2105.05233 (2021). [\ncode\n]\n[9] Jonathan Ho & Tim Salimans.\n“Classifier-Free Diffusion Guidance.”\nNeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.\n[10] Yang Song, et al.\n“Score-Based Generative Modeling through Stochastic Differential Equations.”\nICLR 2021.\n[11] Alex Nichol, Prafulla Dhariwal & Aditya Ramesh, et al.\n“GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.”\nICML 2022.\n[12] Jonathan Ho, et al.\n“Cascaded diffusion models for high fidelity image generation.”\nJ. Mach. Learn. Res. 23 (2022): 47-1.\n[13] Aditya Ramesh et al.\n“Hierarchical Text-Conditional Image Generation with CLIP Latents.”\narxiv Preprint arxiv:2204.06125 (2022).\n[14] Chitwan Saharia & William Chan, et al.\n“Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.”\narxiv Preprint arxiv:2205.11487 (2022).\n[15] Rombach & Blattmann, et al.\n“High-Resolution Image Synthesis with Latent Diffusion Models.”\nCVPR 2022.\ncode\n[16] Song et al.\n“Consistency Models”\narxiv Preprint arxiv:2303.01469 (2023)\n[17] Salimans & Ho.\n“Progressive Distillation for Fast Sampling of Diffusion Models”\nICLR 2022.\n[18] Ronneberger, et al.\n“U-Net: Convolutional Networks for Biomedical Image Segmentation”\nMICCAI 2015.\n[19] Peebles & Xie.\n“Scalable diffusion models with transformers.”\nICCV 2023.\n[20] Zhang et al.\n“Adding Conditional Control to Text-to-Image Diffusion Models.”\narxiv Preprint arxiv:2302.05543 (2023).",
  "metadata": {
    "title": "What are Diffusion Models? | Lil'Log",
    "description": "[Updated on 2021-09-19: Highly recommend this blog post on score-based generative modeling by Yang Song (author of several key papers in the references)].\n[Updated on 2022-08-27: Added classifier-free guidance, GLIDE, unCLIP and Imagen.\n[Updated on 2022-08-31: Added latent diffusion model.\n[Updated on 2024-04-13: Added progressive distillation, consistency models, and the Model Architecture section.",
    "keywords": "generative-model, math-heavy, image-generation",
    "author": "Lilian Weng",
    "og:title": "What are Diffusion Models?",
    "og:description": "[Updated on 2021-09-19: Highly recommend this blog post on score-based generative modeling by Yang Song (author of several key papers in the references)].\n[Updated on 2022-08-27: Added classifier-free guidance, GLIDE, unCLIP and Imagen.\n[Updated on 2022-08-31: Added latent diffusion model.\n[Updated on 2024-04-13: Added progressive distillation, consistency models, and the Model Architecture section.",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/",
    "twitter:card": "summary",
    "twitter:title": "What are Diffusion Models?",
    "twitter:description": "[Updated on 2021-09-19: Highly recommend this blog post on score-based generative modeling by Yang Song (author of several key papers in the references)].\n[Updated on 2022-08-27: Added classifier-free guidance, GLIDE, unCLIP and Imagen.\n[Updated on 2022-08-31: Added latent diffusion model.\n[Updated on 2024-04-13: Added progressive distillation, consistency models, and the Model Architecture section.",
    "article:section": "posts",
    "article:published_time": "2021-07-11T00:00:00+00:00",
    "article:modified_time": "2021-07-11T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/93e53a97f2e3/code/body-000.txt",
      "chars": 253,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/generative-overview.png",
      "bytes": 525595,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-41a09a8696808ecce0b1.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/generative-overview.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/DDPM.png",
      "bytes": 839600,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-e78e0ba4b34cda911aa0.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/DDPM.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/diffusion-example.png",
      "bytes": 4019404,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-9f58f5472b55d74decc4.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/diffusion-example.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/DDPM-algo.png",
      "bytes": 280949,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-8f6527c6c3694c587d18.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/DDPM-algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/diffusion-beta.png",
      "bytes": 128389,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-ea3831d82b483995f8de.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/diffusion-beta.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/improved-DDPM-nll.png",
      "bytes": 461423,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-2a1b7652ba6cc48a1275.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/improved-DDPM-nll.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/conditioned-DDPM.png",
      "bytes": 1535460,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-457cd3bc047052c18add.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/conditioned-DDPM.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/DDIM-results.png",
      "bytes": 136690,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-7b1d616dab0a8609f18b.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/DDIM-results.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/progressive-distillation.png",
      "bytes": 139161,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-feaad52a803d77f9d89b.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/progressive-distillation.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/progressive-distillation-algo.png",
      "bytes": 337076,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-d3eae5a72206efa4852f.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/progressive-distillation-algo.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/consistency-models.png",
      "bytes": 531125,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-f63895c8c6c047926f5f.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/consistency-models.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/consistency-models-exp.png",
      "bytes": 701014,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-cf193ab727b4044922a6.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/consistency-models-exp.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/image-distortion-rate.png",
      "bytes": 858306,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-7865196e2489fc001d22.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/image-distortion-rate.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/latent-diffusion-arch.png",
      "bytes": 338942,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-364a0fd3015ab3fcaca4.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/latent-diffusion-arch.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/cascaded-diffusion.png",
      "bytes": 428232,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-34e80819222058397d52.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/cascaded-diffusion.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/unCLIP.png",
      "bytes": 750479,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-92f9487fb8e9f97c5122.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/unCLIP.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/U-net.png",
      "bytes": 173159,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-7726aba941acd21dd196.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/U-net.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/ControlNet.png",
      "bytes": 96971,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-7ca77e5b5a81baabf985.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/ControlNet.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/DiT.png",
      "bytes": 106505,
      "reused": false,
      "local_path": "attachments/93e53a97f2e3/images/body-b56eeacd975c4881227f.png",
      "src": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/DiT.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://yang-song.github.io/blog/2021/score/",
      "label": "score-based generative modeling",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2017-08-20-gan/",
      "label": "GAN",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-08-12-vae/",
      "label": "VAE",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-10-13-flow-models/",
      "label": "Flow-based",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1503.03585",
      "label": "Sohl-Dickstein et al., 2015",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1907.05600",
      "label": "Yang & Ermon, 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.11239",
      "label": "Ho et al. 2020",
      "type": "a"
    },
    {
      "href": "https://www.stats.ox.ac.uk/~teh/research/compstats/WelTeh2011a.pdf",
      "label": "Welling & Teh 2011",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence",
      "label": "closed form",
      "type": "a"
    },
    {
      "href": "http://www.iro.umontreal.ca/~vincentp/Publications/smdae_techreport.pdf",
      "label": "Vincent, 2011",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1905.07088",
      "label": "Song et al., 2019",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2102.09672",
      "label": "Nichol & Dhariwal (2021)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2105.05233",
      "label": "Dhariwal & Nichol (2021)",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=qw8AKxfYbI",
      "label": "Ho & Salimans, 2021",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2112.10741",
      "label": "Nichol, Dhariwal & Ramesh, et al. 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.02502",
      "label": "Song et al. (2020)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2202.00512",
      "label": "Salimans & Ho, 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2303.01469",
      "label": "Song et al. 2023",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Ordinary_differential_equation",
      "label": "ODE",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2018-02-19-rl-overview/",
      "label": "DQN",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2021-05-31-contrastive/",
      "label": "momentum",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1801.03924",
      "label": "LPIPS",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2112.10752",
      "label": "Rombach & Blattmann, et al. 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2106.15282",
      "label": "Ho et al. (2021)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2204.06125",
      "label": "Ramesh et al. 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2205.11487",
      "label": "Saharia et al. 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1505.04597",
      "label": "Ronneberger, et al. 2015",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.05543",
      "label": "Zhang et al. 2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2212.09748",
      "label": "Peebles & Xie, 2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2006.09011",
      "label": "“Improved techniques for training score-based generative models.”",
      "type": "a"
    },
    {
      "href": "https://github.com/hojonathanho/diffusion",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://github.com/ermongroup/ddim",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/improved-diffusion",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://github.com/openai/guided-diffusion",
      "label": "code",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2207.12598",
      "label": "“Classifier-Free Diffusion Guidance.”",
      "type": "a"
    },
    {
      "href": "https://openreview.net/forum?id=PxTIG12RRHS",
      "label": "“Score-Based Generative Modeling through Stochastic Differential Equations.”",
      "type": "a"
    },
    {
      "href": "https://github.com/CompVis/latent-diffusion",
      "label": "code",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:12Z",
  "source_url": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1503.03585",
      "bytes": 4355107,
      "reused": false,
      "local_path": "papers/repair/cdd3500c06ab19cbba99.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1907.05600",
      "bytes": 8113050,
      "reused": false,
      "local_path": "papers/repair/3b4fd27642b43d30aa2f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.11239",
      "bytes": 10267274,
      "reused": false,
      "local_path": "papers/repair/f8a0cafabed0cfe88ca6.pdf"
    },
    {
      "status": "ok",
      "url": "https://www.stats.ox.ac.uk/~teh/research/compstats/WelTeh2011a.pdf",
      "bytes": 329916,
      "reused": false,
      "local_path": "papers/repair/bbc40d9999e1f9f5fc48.pdf"
    },
    {
      "status": "invalid_file_type",
      "url": "http://www.iro.umontreal.ca/~vincentp/Publications/smdae_techreport.pdf",
      "content_type": "text/html; charset=utf-8",
      "retried_at": "2026-10-20T17:54:23Z"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1905.07088",
      "bytes": 6419033,
      "reused": false,
      "local_path": "papers/repair/6686aebbd982d6117171.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2102.09672",
      "bytes": 12442086,
      "reused": false,
      "local_path": "papers/repair/e655aaac9c8d5692cbb1.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2105.05233",
      "bytes": 39795106,
      "reused": false,
      "local_path": "papers/repair/2cc7d9f8c42587f06289.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2112.10741",
      "bytes": 21259715,
      "reused": false,
      "local_path": "papers/repair/761e685eac3e57a5db36.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.02502",
      "bytes": 10857218,
      "reused": true,
      "local_path": "papers/repair/2e617588af84373a622f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2202.00512",
      "bytes": 3488589,
      "reused": true,
      "local_path": "papers/repair/353b5d2204f89a0a3e98.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2303.01469",
      "bytes": 31529980,
      "reused": true,
      "local_path": "papers/repair/c129e904a22d86f9e0a4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1801.03924",
      "bytes": 5543537,
      "reused": false,
      "local_path": "papers/repair/ff24fdb2d1004cfd5df4.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2112.10752",
      "bytes": 40842586,
      "reused": true,
      "local_path": "papers/repair/84de7ccc1cc12ec4cb29.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2106.15282",
      "bytes": 29348312,
      "reused": false,
      "local_path": "papers/repair/2e72edcbfd718e17ae5f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2204.06125",
      "bytes": 42905303,
      "reused": true,
      "local_path": "papers/repair/037740aca2c3ab83f801.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2205.11487",
      "bytes": 11365725,
      "reused": true,
      "local_path": "papers/repair/8c48830dcf885dd9be79.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1505.04597",
      "bytes": 1648684,
      "reused": false,
      "local_path": "papers/repair/a930a1f7a86248051476.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.05543",
      "bytes": 16345157,
      "reused": false,
      "local_path": "papers/repair/d5ad0d4238973bc696c9.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2212.09748",
      "bytes": 43824062,
      "reused": true,
      "local_path": "papers/repair/6ff6a352b0138ad5d782.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2006.09011",
      "bytes": 13161339,
      "reused": false,
      "local_path": "papers/repair/c2644ba086fa34d7dffc.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2207.12598",
      "bytes": 3773509,
      "reused": false,
      "local_path": "papers/repair/b118a9b962027e6555c7.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 54348,
  "expected_body_images": 19,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "invalid_file_type",
  "raw_html": "attachments/93e53a97f2e3/pages/source.html",
  "local_html": "attachments/93e53a97f2e3/pages/article.html",
  "body_base_path": "attachments/93e53a97f2e3/pages/",
  "body_markdown_path": "attachments/93e53a97f2e3/pages/article.md"
}