{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Lilian Weng / Lil'Log",
  "title_listed": "Diffusion Models for Video Generation",
  "keywords": "视频生成,扩散模型,Sora",
  "domain": "图像处理",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/",
  "publish_date": "2024-04",
  "corpus_stage": "预训练/SFT",
  "notes": "视频生成扩散模型综述",
  "id": "38eb95b0e9e6",
  "collected_at": "2026-10-02T14:12:39",
  "title": "Diffusion Models for Video Generation | Lil'Log",
  "body_markdown": "Diffusion models\nhave demonstrated strong results on image synthesis in past years. Now the research community has started working on a harder task—using it for video generation. The task itself is a superset of the image case, since an image is a video of 1 frame, and it is much more challenging because:\nIt has extra requirements on temporal consistency across frames in time, which naturally demands more world knowledge to be encoded into the model.\nIn comparison to text or images, it is more difficult to collect large amounts of high-quality, high-dimensional video data, let along text-video pairs.\n🥑 Required Pre-read: Please make sure you have read the previous blog on\n“What are Diffusion Models?”\nfor image generation before continue here.\nVideo Generation Modeling from Scratch\n#\nFirst let’s review approaches for designing and training diffusion video models from scratch, meaning that we do not rely on pre-trained image generators.\nParameterization & Sampling Basics\n#\nHere we use a slightly different variable definition from the\nprevious post\n, but the math stays the same. Let $\\mathbf{x} \\sim q_\\text{real}$ be a data point sampled from the real data distribution. Now we are adding Gaussian noise in small amount in time, creating a sequence of noisy variations of $\\mathbf{x}$, denoted as $\\{\\mathbf{z}_t \\mid t =1 \\dots, T\\}$, with increasing amount of noise as $t$ increases and the last $q(\\mathbf{z}_T) \\sim \\mathcal{N}(\\mathbf{0}, \\mathbf{I})$. The noise-adding forward process is a Gaussian process. Let $\\alpha_t, \\sigma_t$ define a differentiable noise schedule of the Gaussian process:\n$$\nq(\\mathbf{z}_t \\vert \\mathbf{x}) = \\mathcal{N}(\\mathbf{z}_t; \\alpha_t \\mathbf{x}, \\sigma^2_t\\mathbf{I})\n$$\nTo represent $q(\\mathbf{z}_t \\vert \\mathbf{z}_s)$ for $0 \\leq s < t \\leq T$, we have:\n$$\n\\begin{aligned}\n\\mathbf{z}_t &= \\alpha_t \\mathbf{x} + \\sigma_t\\boldsymbol{\\epsilon}_t \\\\\n\\mathbf{z}_s &= \\alpha_s \\mathbf{x} + \\sigma_s\\boldsymbol{\\epsilon}_s \\\\\n\\mathbf{z}_t &= \\alpha_t \\Big(\\frac{\\mathbf{z}_s - \\sigma_s\\boldsymbol{\\epsilon}_s}{\\alpha_s}\\Big) + \\sigma_t\\boldsymbol{\\epsilon}_t \\\\\n\\mathbf{z}_t &= \\frac{\\alpha_t}{\\alpha_s}\\mathbf{z}_s + \\sigma_t\\boldsymbol{\\epsilon}_t - \\frac{\\alpha_t\\sigma_s}{\\alpha_s} \\boldsymbol{\\epsilon}_s \\\\\n\\text{Thus }q(\\mathbf{z}_t \\vert \\mathbf{z}_s) &= \\mathcal{N}\\Big(\\mathbf{z}_t; \\frac{\\alpha_t}{\\alpha_s}\\mathbf{z}_s, \\big(1 - \\frac{\\alpha^2_t\\sigma^2_s}{\\sigma^2_t\\alpha^2_s}\\big)\\sigma^2_t \\mathbf{I}\\Big)\n\\end{aligned}\n$$\nLet the log signal-to-noise-ratio be $\\lambda_t = \\log[\\alpha^2_t / \\sigma^2_t]$, we can represent the DDIM (\nSong et al. 2020\n) update as:\n$$\nq(\\mathbf{z}_t \\vert \\mathbf{z}_s) = \\mathcal{N}\\Big(\\mathbf{z}_t; \\frac{\\alpha_t}{\\alpha_s}\\mathbf{z}_s, \\sigma^2_{t\\vert s} \\mathbf{I}\\Big) \\quad\n\\text{where }\\sigma^2_{t\\vert s} = (1 - e^{\\lambda_t - \\lambda_s})\\sigma^2_t\n$$\nThere is a special $\\mathbf{v}$-prediction ($\\mathbf{v} = \\alpha_t \\boldsymbol{\\epsilon} - \\sigma_t \\mathbf{x}$) parameterization, proposed by\nSalimans & Ho (2022)\n. It has been shown to be helpful for avoiding color shift in video generation compared to $\\boldsymbol{\\epsilon}$-parameterization.\nThe $\\mathbf{v}$-parameterization is derived with a trick in the angular coordinate. First, we define $\\phi_t = \\arctan(\\sigma_t / \\alpha_t)$ and thus we have $\\alpha_\\phi = \\cos\\phi, \\sigma_t = \\sin\\phi, \\mathbf{z}_\\phi = \\cos\\phi \\mathbf{x} + \\sin\\phi\\boldsymbol{\\epsilon}$. The velocity of $\\mathbf{z}_\\phi$ can be written as:\n$$\n\\mathbf{v}_\\phi = \\nabla_\\phi \\mathbf{z}_\\phi = \\frac{d\\cos\\phi}{d\\phi} \\mathbf{x} + \\frac{d\\sin\\phi}{d\\phi}\\boldsymbol{\\epsilon} = \\cos\\phi\\boldsymbol{\\epsilon} -\\sin\\phi\\mathbf{x}\n$$\nThen we can infer,\n$$\n\\begin{aligned}\n\\sin\\phi\\mathbf{x} \n&= \\cos\\phi\\boldsymbol{\\epsilon}  - \\mathbf{v}_\\phi \\\\\n&= \\frac{\\cos\\phi}{\\sin\\phi}\\big(\\mathbf{z}_\\phi - \\cos\\phi\\mathbf{x}\\big) - \\mathbf{v}_\\phi \\\\\n\\sin^2\\phi\\mathbf{x} \n&= \\cos\\phi\\mathbf{z}_\\phi - \\cos^2\\phi\\mathbf{x} - \\sin\\phi \\mathbf{v}_\\phi \\\\\n\\mathbf{x} &= \\cos\\phi\\mathbf{z}_\\phi - \\sin\\phi\\mathbf{v}_\\phi \\\\\n\\text{Similarly }\n\\boldsymbol{\\epsilon} &= \\sin\\phi\\mathbf{z}_\\phi + \\cos\\phi \\mathbf{v}_\\phi\n\\end{aligned}\n$$\nThe DDIM update rule is updated accordingly,\n$$\n\\begin{aligned}\n\\mathbf{z}_{\\phi_s} \n&= \\cos\\phi_s\\hat{\\mathbf{x}}_\\theta(\\mathbf{z}_{\\phi_t}) + \\sin\\phi_s\\hat{\\epsilon}_\\theta(\\mathbf{z}_{\\phi_t}) \\quad\\quad{\\small \\text{; }\\hat{\\mathbf{x}}_\\theta(.), \\hat{\\epsilon}_\\theta(.)\\text{ are two models to predict }\\mathbf{x}, \\boldsymbol{\\epsilon}\\text{ based on }\\mathbf{z}_{\\phi_t}}\\\\\n&= \\cos\\phi_s \\big( \\cos\\phi_t \\mathbf{z}_{\\phi_t} - \\sin\\phi_t \\hat{\\mathbf{v}}_\\theta(\\mathbf{z}_{\\phi_t} ) \\big) +\n\\sin\\phi_s \\big( \\sin\\phi_t \\mathbf{z}_{\\phi_t} + \\cos\\phi_t \\hat{\\mathbf{v}}_\\theta(\\mathbf{z}_{\\phi_t} ) \\big) \\\\\n&= {\\color{red} \\big( \\cos\\phi_s\\cos\\phi_t + \\sin\\phi_s\\sin\\phi_t \\big)} \\mathbf{z}_{\\phi_t} + \n{\\color{green} \\big( \\sin\\phi_s \\cos\\phi_t - \\cos\\phi_s \\sin\\phi_t \\big)} \\hat{\\mathbf{v}}_\\theta(\\mathbf{z}_{\\phi_t} ) \\\\\n&= {\\color{red} cos(\\phi_s - \\phi_t)} \\mathbf{z}_{\\phi_t} +\n{\\color{green} \\sin(\\phi_s - \\phi_t)} \\hat{\\mathbf{v}}_\\theta(\\mathbf{z}_{\\phi_t}) \\quad\\quad{\\small \\text{; trigonometric identity functions.}}\n\\end{aligned}\n$$\nVisualizing how the diffusion update step works in the angular coordinate, where DDIM evolves $\\mathbf{z}_{\\phi_s}$ by moving it along the $-\\hat{\\mathbf{v}}_{\\phi_t}$ direction. (Image source:\nSalimans & Ho, 2022\n)\nThe $\\mathbf{v}$-parameterization for the model is to predict $\\mathbf{v}_\\phi = \\cos\\phi\\boldsymbol{\\epsilon} -\\sin\\phi\\mathbf{x} = \\alpha_t\\boldsymbol{\\epsilon} - \\sigma_t\\mathbf{x}$.\nIn the case of video generation, we need the diffusion model to run multiple steps of upsampling for extending video length or increasing the frame rate. This requires the capability of sampling a second video $\\mathbf{x}^b$ conditioned on the first $\\mathbf{x}^a$, $\\mathbf{x}^b \\sim p_\\theta(\\mathbf{x}^b \\vert \\mathbf{x}^a)$, where $\\mathbf{x}^b$ might be an autoregressive extension of $\\mathbf{x}^a$ or be the missing frames in-between for a video $\\mathbf{x}^a$ at a low frame rate.\nThe sampling of $\\mathbf{x}_b$ needs to condition on $\\mathbf{x}_a$ besides its own corresponding noisy variable.\nVideo Diffusion Models\n(\nVDM\n;\nHo & Salimans, et al. 2022\n) proposed the\nreconstruction guidance\nmethod using an adjusted denoising model such that the sampling of $\\mathbf{x}^b$ can be properly conditioned on $\\mathbf{x}^a$:\n$$\n\\begin{aligned}\n\\mathbb{E}_q [\\mathbf{x}_b \\vert \\mathbf{z}_t, \\mathbf{x}^a] &= \\mathbb{E}_q [\\mathbf{x}^b \\vert \\mathbf{z}_t] + \\frac{\\sigma_t^2}{\\alpha_t} \\nabla_{\\mathbf{z}^b_t} \\log q(\\mathbf{x}^a \\vert \\mathbf{z}_t) \\\\\nq(\\mathbf{x}^a \\vert \\mathbf{z}_t) &\\approx \\mathcal{N}\\big[\\hat{\\mathbf{x}}^a_\\theta (\\mathbf{z}_t), \\frac{\\sigma_t^2}{\\alpha_t^2}\\mathbf{I}\\big] & {\\small \\text{; the closed form is unknown.}}\\\\\n\\tilde{\\mathbf{x}}^b_\\theta (\\mathbf{z}_t) &= \\hat{\\mathbf{x}}^b_\\theta (\\mathbf{z}_t) - \\frac{w_r \\alpha_t}{2} \\nabla_{\\mathbf{z}_t^b} \\| \\mathbf{x}^a - \\hat{\\mathbf{x}}^a_\\theta (\\mathbf{z}_t) \\|^2_2 & {\\small \\text{; an adjusted denoising model for }\\mathbf{x}^b}\n\\end{aligned}\n$$\nwhere $\\hat{\\mathbf{x}}^a_\\theta (\\mathbf{z}_t), \\hat{\\mathbf{x}}^b_\\theta (\\mathbf{z}_t)$ are reconstructions of $\\mathbf{x}^a, \\mathbf{x}^b$ provided by the denoising model. And $w_r$ is a weighting factor and a large one $w_r >1$ is found to improve sample quality. Note that it is also possible to simultaneously condition on low resolution videos to extend samples to be at the high resolution using the same reconstruction guidance method.\nModel Architecture: 3D U-Net & DiT\n#\nSimilar to text-to-image diffusion models, U-net and Transformer are still two\ncommon architecture choices\n. There are a series of diffusion video modeling papers from Google based on the U-net architecture and a recent Sora model from OpenAI leveraged the Transformer architecture.\nVDM\n(\nHo & Salimans, et al. 2022\n) adopts the standard diffusion model setup but with an altered architecture suitable for video modeling. It extends the\n2D U-net\nto work for 3D data (\nCicek et al. 2016\n), where each feature map represents a 4D tensor of frames x height x width x channels. This 3D U-net is factorized over space and time, meaning that each layer only operates on the space or time dimension, but not both:\nProcessing\nSpace\n:\nEach old 2D convolution layer as in the 2D U-net is extended to be space-only 3D convolution; precisely, 3x3 convolutions become 1x3x3 convolutions.\nEach spatial attention block remains as attention over space, where the first axis (\nframes\n) is treated as batch dimension.\nProcessing\nTime\n:\nA temporal attention block is added after each spatial attention block. It performs attention over the first axis (\nframes\n) and treats spatial axes as the batch dimension. The\nrelative position\nembedding is used for tracking the order of frames. The temporal attention block is important for the model to capture good temporal coherence.\nThe 3D U-net architecture. The noisy video $\\mathbf{z}_t$ , conditioning information $\\boldsymbol{c}$ and the log signal-to-noise ratio (log-SNR) $\\lambda_t$ are inputs to the network. The channel multipliers $M_1, \\dots, M_K$ represent the channel counts across layers. (Image source:\nSalimans & Ho, 2022\n)\nImagen Video\n(\nHo, et al. 2022\n) is constructed on a cascade of diffusion models to enhance the video generation quality and upgrades to output 1280x768 videos at 24 fps. The Imagen Video architecture consists of the following components, counting 7 diffusion models in total.\nA frozen\nT5\ntext encoder to provide text embedding as the conditioning input.\nA base video diffusion model.\nA cascade of interleaved\nspatial and temporal super-resolution\ndiffusion models, including 3 TSR (Temporal Super-Resolution) and 3 SSR (Spatial Super-Resolution) components.\nThe cascaded sampling pipeline in Imagen Video. In practice, the text embeddings are injected into all components, not just the base model. (Image source:\nHo et al. 2022\n)\nThe base denoising models performs spatial operations over all the frames with shared parameters simultaneously and then the temporal layer mixes activations across frames to better capture temporal coherence, which is found to work better than frame-autoregressive approaches.\nThe architecture of one space-time separable block in the Imagen Video diffusion model. (Image source:\nHo et al. 2022\n)\nBoth SSR and TSR models condition on the upsampled inputs concatenated with noisy data $\\mathbf{z}_t$ channel-wise. SSR upsamples by\nbilinear resizing\n, while TSR upsamples by repeating the frames or filling in blank frames.\nImagen Video also applies\nprogressive distillation\nto speed up sampling and each distillation iteration can reduce the required sampling steps by half. Their experiments were able to distill all 7 video diffusion models down to just 8 sampling steps per model without any noticeable loss in perceptual quality.\nTo achieve better scaling efforts,\nSora\n(\nBrooks et al. 2024\n) leverages\nDiT (Diffusion Transformer)\narchitecture that operates on spacetime patches of video and image latent codes. Visual input is represented as a sequence of spacetime patches which act as Transformer input tokens.\nSora is a diffusion transformer model.\n(Image source:\nBrooks et al. 2024\n)\nAdapting Image Models to Generate Videos\n#\nAnother prominent approach for diffusion video modeling is to “inflate” a pre-trained image-to-text diffusion model by inserting temporal layers and then we can choose to\nonly\nfine-tune new layers on video data, or avoid extra training at all. The prior knowledge of text-image pairs is inherited by the new model and thus it can help alleviate the requirement on text-video pair data.\nFine-tuning on Video Data\n#\nMake-A-Video\n(\nSinger et al. 2022\n) extends a pre-trained diffusion image model with a temporal dimension, consisting of three key components:\nA base text-to-image model trained on text-image pair data.\nSpatiotemporal convolution and attention layers to extend the network to cover temporal dimension.\nA frame interpolation network for high frame rate generation\nThe illustration of Make-A-Video pipeline.\n(Image source:\nSinger et al. 2022\n)\nThe final video inference scheme can be formulated as:\n$$\n\\hat{\\mathbf{y}}_t = \\text{SR}_h \\circ \\text{SR}^t_l \\circ \\uparrow_F \\circ D^t \\circ P \\circ (\\hat{\\mathbf{x}}, \\text{CLIP}_\\text{text}(\\mathbf{x}))\n$$\nwhere:\n$\\mathbf{x}$ is the input text.\n$\\hat{\\mathbf{x}}$ is the BPE-encoded text.\n$\\text{CLIP}_\\text{text}(.)$ is the CLIP text encoder, $\\mathbf{x}_e = \\text{CLIP}_\\text{text}(\\mathbf{x})$.\n$P(.)$ is the prior, generating image embedding $\\mathbf{y}_e$ given text embedding $\\mathbf{x}_e$ and BPE encoded text $\\hat{\\mathbf{x}}$ : $\\mathbf{y}_e = P(\\mathbf{x}_e, \\hat{\\mathbf{x}})$. This part is trained on text-image pair data and not fine-tuned on video data.\n$D^t(.)$ is the spatiotemporal decoder that generates a series of 16 frames, where each frame is a low-resolution 64x64 RGB image $\\hat{\\mathbf{y}}_l$.\n$\\uparrow_F(.)$ is the frame interpolation network, increasing the effective frame rate by interpolating between generated frames. This is a fine-tuned model for the task of predicting masked frames for video upsampling.\n$\\text{SR}_h(.), \\text{SR}^t_l(.)$ are the spatial and spatiotemporal super-resolution models, increasing the image resolution to 256x256 and 768x768, respectively.\n$\\hat{\\mathbf{y}}_t$ is the final generated video.\nSpatiotemporal SR layers contain pseudo-3D convo layers and pseudo-3D attention layers:\nPseudo-3D convo layer : Each spatial 2D convo layer (initialized from the pre-training image model) is followed by a temporal 1D layer (initialized as the identity function). Conceptually, the convo 2D layer first generates multiple frames and then frames are reshaped to be a video clip.\nPseudo-3D attention layer: Following each (pre-trained) spatial attention layer, a temporal attention layer is stacked and used to approximate a full spatiotemporal attention layer.\nHow pseudo-3D convolution (left) and attention (right) layers work.\n(Image source:\nSinger et al. 2022\n)\nThey can be represented as:\n$$\n\\begin{aligned}\n\\text{Conv}_\\text{P3D} &= \\text{Conv}_\\text{1D}(\\text{Conv}_\\text{2D}(\\mathbf{h}) \\circ T) \\circ T \\\\\n\\text{Attn}_\\text{P3D} &= \\text{flatten}^{-1}(\\text{Attn}_\\text{1D}(\\text{Attn}_\\text{2D}(\\text{flatten}(\\mathbf{h})) \\circ T) \\circ T)\n\\end{aligned}\n$$\nwhere an input tensor $\\mathbf{h} \\in \\mathbb{R}^{B\\times C \\times F \\times H \\times W}$  (corresponding to batch size, channels, frames, height and weight); and $\\circ T$ swaps between temporal and spatial dimensions; $\\text{flatten}(.)$ is a matrix operator to convert $\\mathbf{h}$ to be $\\mathbf{h}’ \\in \\mathbb{R}^{B \\times C \\times F \\times HW}$ and $\\text{flatten}^{-1}(.)$ reverses that process.\nDuring training, different components of Make-A-Video pipeline are trained independently.\nDecoder $D^t$, prior $P$ and two super-resolution components $\\text{SR}_h, \\text{SR}^t_l$ are first trained on images alone, without paired text.\nNext the new temporal layers are added, initialized as identity function, and then fine-tuned on unlabeled video data.\nTune-A-Video\n(\nWu et al. 2023\n) inflates a pre-trained image diffusion model to enable one-shot video tuning: Given a video containing $m$ frames, $\\mathcal{V} = \\{v_i \\mid i = 1, \\dots, m\\}$, paired with a descriptive prompt $\\tau$, the task is to generate a new video $\\mathcal{V}^*$ based on a slightly edited & related text prompt $\\tau^*$. For example, $\\tau$ =\n\"A man is skiing\"\ncan be extended to $\\tau^*$=\n\"Spiderman is skiing on the beach\"\n. Tune-A-Video is meant to be used for object editing, background change, and style transfer.\nBesides inflating the 2D convo layer, the U-Net architecture of Tune-A-Video incorporates the ST-Attention (spatiotemporal attention) block to capture temporal consistency by querying relevant positions in previous frames. Given latent features of frame $v_i$, previous frames $v_{i-1}$ and the first frame $v_1$ are projected to query $\\mathbf{Q}$, key $\\mathbf{K}$ and value $\\mathbf{V}$, the ST-attention is defined as:\n$$\n\\begin{aligned}\n&\\mathbf{Q} = \\mathbf{W}^Q \\mathbf{z}_{v_i}, \\quad \\mathbf{K} = \\mathbf{W}^K [\\mathbf{z}_{v_1}, \\mathbf{z}_{v_{i-1}}], \\quad \\mathbf{V} = \\mathbf{W}^V [\\mathbf{z}_{v_1}, \\mathbf{z}_{v_{i-1}}] \\\\\n&\\mathbf{O} = \\text{softmax}\\Big(\\frac{\\mathbf{Q} \\mathbf{K}^\\top}{\\sqrt{d}}\\Big) \\cdot \\mathbf{V}\n\\end{aligned}\n$$\nThe Tune-A-Video architecture overview. It first runs a light-weighted fine-tuning stage on a single video before the sampling stage. Note that the entire temporal self-attention (T-Attn) layers get fine-tuned because they are newly added, but only query projections in ST-Attn and Cross-Attn are updated during fine-tuning to preserve prior text-to-image knowledge. ST-Attn improves spatial-temporal consistency, Cross-Attn refines text-video alignment. (Image source:\nWu et al. 2023\n)\nGen-1\nmodel  (\nEsser et al. 2023\n) by Runway targets the task of editing a given video according to text inputs. It decomposes the consideration of\nstructure\nand\ncontent\nof a video $p(\\mathbf{x} \\mid s, c)$ for generation conditioning. However, to do a clear decomposition of these two aspects is not easy.\nContent\n$c$ refers to appearance and semantics of the video, that is sampled from the text for conditional editing. CLIP embedding of the frame is a good representation of content, and stays largely orthogonal to structure traits.\nStructure\n$s$ depicts greometry and dynamics, including shapes, locations, temporal changes of objects, and $s$ is sampled from the input video. Depth estimation or other task-specific side information (e.g. human body pose or face landmarks for human video synthesis) can be used.\nThe architecture changes in Gen-1 are quite standard, i.e. adding 1D temporal convo layer after each 2D spatial convo layer in its residual blocks and adding 1D temporal attention block after each 2D spatial attention block in its attention blocks. During training, the structure variable $s$ is concatenated with the diffusion latent variable $\\mathbf{z}$, where the content variable $c$ is provided in the cross-attention layer. At inference time, the clip embedding is converted via a prior to convert CLIP text embedding to be CLIP image embedding.\nThe overview of the Gen-1 model training pipeline.\n(Image source:\nEsser et al. 2023\n)\nVideo LDM\n(\nBlattmann et al. 2023\n) trains a\nLDM\n(Latent diffusion models) image generator first. Then the model is fine-tuned to produce videos with a temporal dimension added. The fine-tuning only applies to these newly added temporal layers on encoded image sequences. The temporal layers $\\{l^i_\\phi \\mid i = \\ 1, \\dots, L\\}$ in the Video LDM (See Fig. 10) are interleaved with existing spatial layers $l^i_\\theta$ which stays\nfrozen\nduring fine-tuning. That’s being said, we only fine-tune the new parameters $\\phi$ but not the pre-trained image backbone model parameters $\\theta$. The pipeline of Video LDM first generates key frames at low fps and then processes through 2 steps of latent frame interpolations to increase fps.\nThe input sequence of length $T$ is interpreted as a batch of images (i.e. $B \\cdot T$) for the base image model $\\theta$ and then gets reshaped into video format for $l^i_\\phi$ temporal layers. There is a skip connection leads to a combination of temporal layer output $\\mathbf{z}’$ and the spatial output $\\mathbf{z}$ via a learned merging parameter $\\alpha$. There are two types of temporal mixing layers implemented in practice: (1) temporal attention and (2) residual blocks based on 3D convolutions.\nA pre-training LDM for image synthesis is extended to be a video generator. $B, T, C, H, W$ are batch size, sequence length, channels, height and width, respectively. $\\mathbf{c}_S$ is an optional conditioning/context frame. (Image source:\nBlattmann et al. 2023\n)\nHowever, there is a remaining issue with LDM’s pretrainined autoencoder which only sees images never videos. Naively using that for video generation can cause flickering artifacts without good temporal coherence. So Video LDM adds additional temporal layers into the decoder and fine-tuned on video data with a patch-wise temporal discriminator built from 3D convolutions, while the encoder remains unchanged so that we still can reuse the pretrained LDM. During temporal decoder fine-tuning, the frozen encoder processes each frame in the video independently, and enforce temporally coherent reconstructions across frames with a video-aware discriminator.\nThe training pipeline of autoencoder in video latent diffusion models. The decoder is fine-tuned to have temporal coherency with a new across-frame discriminator while the encoder stays frozen. (Image source:\nBlattmann et al. 2023\n)\nSimilar to Video LDM, the architecture design of\nStable Video Diffusion\n(\nSVD\n;\nBlattmann et al. 2023\n) is also based on LDM with temporal layers inserted after every spatial convolution and attention layer, but SVD fine-tunes the entire model. There are three stages for training video LDMs:\nText-to-image pretraining\nis important and helps improve both quality and prompt following.\nVideo pretraining\nis beneficial to be separated and should ideally occur on a larger scale, curated dataset.\nHigh-quality video finetuning\nworks with a smaller, pre-captioned video of high visual fidelity.\nSVD specially emphasizes the critical role of\ndataset curation\nin model performance. They applied a cut detection pipeline to get more cuts per video and then applied three different captioner models: (1) CoCa for mid-frame, (2) V-BLIP for a video caption, and (3) LLM based captioning based on previous two captions. Then they were able to continue to improve video datasets, by removing clips with less motion (filtered by low optical flow scores calculated at 2 fps), excessive text presence (apply optical character recognition to identify videos with lots of text), or generally low aesthetic value (annotate the first, middle, and last frames of each clip with CLIP embeddings and calculate aesthetics scores & text-image similarities). The experiments showed that a filtered, higher quality dataset leads to better model quality, even when this dataset is much smaller.\nThe key challenge of generating distant key frames first and then adding interpolation with temporal super-resolution is how to maintain high-quality temporal consistency.\nLumiere\n(\nBar-Tal et al. 2024\n) instead adopts a\nspace-time U-Net (STUNet)\narchitecture that generates the entire temporal duration of the video\nat once\nthrough a single pass, removing the dependency on TSR (temporal super-resolution) components. STUNet downsamples the video in both time and space dimensions and thus expensive computation happens in a compact time-space latent space.\nLumiere removes TSR (temporal super-resolution) models. The inflated SSR network can operate only on short segments of the video due to memory constraints and thus SSR models operate on a set of shorter but overlapped video snippets. (Image source:\nBar-Tal et al. 2024\n)\nSTUNet inflates a\npretrained\ntext-to-image U-net to be able to downsample and upsample videos at both time and space dimensions. Convo-based blocks consist of pre-trained text-to-image layers, followed by a factorized space-time convolution. And attention-based blocks at the coarsest U-Net level contains the pre-trained text-to-image, followed by temporal attention. Further training\nonly\nhappens with the newly added layers.\nThe architecture of (a) Space-Time U-Net (STUNet), (b) the convolution-based block, and (c) the attention-based block. (Image source:\nBar-Tal et al. 2024\n)\nTraining-Free Adaptation\n#\nSomehow surprisingly, it is possible to adapt a pre-trained text-to-image model to output videos without any training 🤯.\nIf we naively sample a sequence of latent codes at random and then construct a video of decoded corresponding images, there is no guarantee in the consistency in objects and semantics in time.\nText2Video-Zero\n(\nKhachatryan et al. 2023\n) enables zero-shot, training-free video generation by enhancing a pre-trained image diffusion model with two key mechanisms for temporal consistency:\nSampling the sequence of latent codes with\nmotion dynamics\nto keep the global scene and the background time consistent;\nReprogramming frame-level self-attention using a\nnew cross-frame attention\nof each frame on the first frame, to preserve the context, appearance, and identity of the foreground object.\nAn overview of the Text2Video-Zero pipeline. (Image source:\nKhachatryan et al. 2023\n)\nThe process of sampling a sequence of latent variables, $\\mathbf{x}^1_T, \\dots, \\mathbf{x}^m_T$, with motion information is described as follows:\nDefine a direction $\\boldsymbol{\\delta} = (\\delta_x, \\delta_y) \\in \\mathbb{R}^2$ for controlling the global scene and camera motion; by default, we set $\\boldsymbol{\\delta} = (1, 1)$. Also define a hyperparameter $\\lambda > 0$ controlling the amount of global motion.\nFirst sample the latent code of the first frame at random, $\\mathbf{x}^1_T \\sim \\mathcal{N}(0, I)$;\nPerform $\\Delta t \\geq 0$ DDIM backward update steps using the pre-trained image diffusion model, e.g. Stable Diffusion (SD) model in the paper, and obtain the corresponding latent code $\\mathbf{x}^1_{T’}$ where $T’ = T - \\Delta t$.\nFor each frame in the latent code sequence, we apply corresponding motion translation with a warping operation defined by $\\boldsymbol{\\delta}^k = \\lambda(k-1)\\boldsymbol{\\delta}$ to obtain $\\tilde{\\mathbf{x}}^k_{T’}$.\nFinally apply DDIM forward steps to all $\\tilde{\\mathbf{x}}^{2:m}_{T’}$ to obtain $\\mathbf{x}^{2:m}_T$.\n$$\n\\begin{aligned}\n\\mathbf{x}^1_{T'} &= \\text{DDIM-backward}(\\mathbf{x}^1_T, \\Delta t)\\text{ where }T' = T - \\Delta t \\\\\nW_k &\\gets \\text{a warping operation of }\\boldsymbol{\\delta}^k = \\lambda(k-1)\\boldsymbol{\\delta} \\\\\n\\tilde{\\mathbf{x}}^k_{T'} &= W_k(\\mathbf{x}^1_{T'})\\\\\n\\mathbf{x}^k_T &= \\text{DDIM-forward}(\\tilde{\\mathbf{x}}^k_{T'}, \\Delta t)\\text{ for }k=2, \\dots, m\n\\end{aligned}\n$$\nBesides, Text2Video-Zero replaces the self-attention layer in a pre-trained SD model with a new cross-frame attention mechanism with reference to the\nfirst\nframe. The motivation is to preserve the information about the foreground object’s appearance, shape, and identity throughout the generated video.\n$$\n\\text{Cross-Frame-Attn}(\\mathbf{Q}^k, \\mathbf{K}^{1:m}, \\mathbf{V}^{1:m}) = \\text{Softmax}\\Big( \\frac{\\mathbf{Q}^k (\\mathbf{K}^1)^\\top}{\\sqrt{c}} \\Big) \\mathbf{V}^1\n$$\nOptionally, the background mask can be used to further smoothen and improve background consistency. Let’s say, we obtain a corresponding foreground mask $\\mathbf{M}_k$ for the $k$-th frame using some existing method, and background smoothing merges the actual and the warped latent code at the diffusion step $t$, w.r.t. the background matrix:\n$$\n\\bar{\\mathbf{x}}^k_t = \\mathbf{M}^k \\odot \\mathbf{x}^k_t + (1 − \\mathbf{M}^k) \\odot (\\alpha\\tilde{\\mathbf{x}}^k_t +(1−\\alpha)\\mathbf{x}^k_t)\\quad\\text{for }k=1, \\dots, m\n$$\nwhere $\\mathbf{x}^k_t$ is the actual latent code and $\\tilde{\\mathbf{x}}^k_t$ is the warped latent code on the background; $\\alpha$ is a hyperparameter and the papers set $\\alpha=0.6$ in the experiments.\nText2video-zero can be combined with\nControlNet\nwhere the ControlNet pretrained copy branch is applied per frame on each $\\mathbf{x}^k_t$ for $k = 1, \\dots, m$ in each diffusion time-step $t = T , \\dots, 1$ and add the ControlNet branch outputs to the skip-connections of the main U-net.\nControlVideo\n(\nZhang et al. 2023\n) aims to generate videos conditioned on text prompt $\\tau$ and a motion sequence (e.g., depth or edge maps), $\\mathbf{c} = \\{c^i\\}_{i=0}^{N-1}$. It is adapted from\nControlNet\nwith three new mechanisms added:\nCross-frame attention\n: Adds fully cross-frame interaction in self-attention modules. It introduces interactions between all the frames, by mapping the latent frames at\nall the time steps\ninto $\\mathbf{Q}, \\mathbf{K}, \\mathbf{V}$ matrices, different from Text2Video-zero which only configures all the frames to attend to the\nfirst\nframe.\nInterleaved-frame smoother\nis a mechanism to employ frame interpolation on alternated frames to reduce the flickering effect. At each time step $t$, the smoother interpolates the even or odd frames to smooth their corresponding three-frame clips. Note that the number of frames decreases in time after smoothing steps.\nHierarchical sampler\nutilizes a hierarchical sampler to enable long videos with time consistency under memory constraints. A long video is split into multiple short clips and each has a key frame selected. The model pre-generates these keyframes with full cross-frame attention for long-term coherency and each corresponding short clip is synthesized sequentially conditioned on the keyframes.\nThe overview of ControlVideo. (Image source:\nZhang et al. 2023\n)\nCitation\n#\nCited as:\nWeng, Lilian. (Apr 2024). Diffusion Models Video Generation. Lil’Log. https://lilianweng.github.io/posts/2024-04-12-diffusion-video/.\nOr\n@article{weng2024video,\n  title   = \"Diffusion Models Video Generation.\",\n  author  = \"Weng, Lilian\",\n  journal = \"lilianweng.github.io\",\n  year    = \"2024\",\n  month   = \"Apr\",\n  url     = \"https://lilianweng.github.io/posts/2024-04-12-diffusion-video/\"\n}\nReferences\n#\n[1] Cicek et al. 2016.\n“3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation.”\n[2] Ho & Salimans, et al.\n“Video Diffusion Models.”\n2022 |\nwebpage\n[3] Bar-Tal et al. 2024\n“Lumiere: A Space-Time Diffusion Model for Video Generation.”\n[4] Brooks et al.\n“Video generation models as world simulators.”\nOpenAI Blog, 2024.\n[5] Zhang et al. 2023\n“ControlVideo: Training-free Controllable Text-to-Video Generation.”\n[6] Khachatryan et al. 2023\n“Text2Video-Zero: Text-to-image diffusion models are zero-shot video generators.”\n[7] Ho, et al. 2022\n“Imagen Video: High Definition Video Generation with Diffusion Models.”\n[8] Singer et al.\n“Make-A-Video: Text-to-Video Generation without Text-Video Data.”\n2022.\n[9] Wu et al.\n“Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation.”\nICCV 2023.\n[10] Blattmann et al. 2023\n“Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models.”\n[11] Blattmann et al. 2023\n“Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets.”\n[12] Esser et al. 2023\n“Structure and Content-Guided Video Synthesis with Diffusion Models.”\n[13] Bar-Tal et al. 2024\n“Lumiere: A Space-Time Diffusion Model for Video Generation.”",
  "metadata": {
    "title": "Diffusion Models for Video Generation | Lil'Log",
    "description": "Diffusion models have demonstrated strong results on image synthesis in past years. Now the research community has started working on a harder task—using it for video generation. The task itself is a superset of the image case, since an image is a video of 1 frame, and it is much more challenging because:\n\nIt has extra requirements on temporal consistency across frames in time, which naturally demands more world knowledge to be encoded into the model.\nIn comparison to text or images, it is more difficult to collect large amounts of high-quality, high-dimensional video data, let along text-video pairs.\n\n\n\n🥑 Required Pre-read: Please make sure you have read the previous blog on “What are Diffusion Models?” for image generation before continue here.",
    "keywords": "generative-model, video-generation",
    "author": "Lilian Weng",
    "og:title": "Diffusion Models for Video Generation",
    "og:description": "Diffusion models have demonstrated strong results on image synthesis in past years. Now the research community has started working on a harder task—using it for video generation. The task itself is a superset of the image case, since an image is a video of 1 frame, and it is much more challenging because:\n\nIt has extra requirements on temporal consistency across frames in time, which naturally demands more world knowledge to be encoded into the model.\nIn comparison to text or images, it is more difficult to collect large amounts of high-quality, high-dimensional video data, let along text-video pairs.\n\n\n\n🥑 Required Pre-read: Please make sure you have read the previous blog on “What are Diffusion Models?” for image generation before continue here.",
    "og:type": "article",
    "og:url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/",
    "twitter:card": "summary",
    "twitter:title": "Diffusion Models for Video Generation",
    "twitter:description": "Diffusion models have demonstrated strong results on image synthesis in past years. Now the research community has started working on a harder task—using it for video generation. The task itself is a superset of the image case, since an image is a video of 1 frame, and it is much more challenging because:\n\nIt has extra requirements on temporal consistency across frames in time, which naturally demands more world knowledge to be encoded into the model.\nIn comparison to text or images, it is more difficult to collect large amounts of high-quality, high-dimensional video data, let along text-video pairs.\n\n\n\n🥑 Required Pre-read: Please make sure you have read the previous blog on “What are Diffusion Models?” for image generation before continue here.",
    "article:section": "posts",
    "article:published_time": "2024-04-12T00:00:00+00:00",
    "article:modified_time": "2024-04-12T00:00:00+00:00"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/38eb95b0e9e6/code/body-000.txt",
      "chars": 256,
      "language": "text"
    }
  ],
  "images": [
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/v-param.png",
      "bytes": 98723,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-d1cf692e1a080f613c29.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/v-param.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/3D-U-net.png",
      "bytes": 68997,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-adcf0e2dd94dc55aa8c6.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/3D-U-net.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/imagen-video.png",
      "bytes": 85205,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-76d8802c2571f9a50ddf.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/imagen-video.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/imagen-video-Unet-block.png",
      "bytes": 126391,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-ff6a78ef36bb37bb9a66.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/imagen-video-Unet-block.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/sora.png",
      "bytes": 755137,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-47830ffccd1f00ebb183.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/sora.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/make-a-video.png",
      "bytes": 2654895,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-ca417bea32b2f2641a57.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/make-a-video.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/make-a-video-layers.png",
      "bytes": 1806248,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-d7659d18c29586c12baf.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/make-a-video-layers.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/tune-a-video.png",
      "bytes": 476898,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-bae332d2c6a1ed910398.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/tune-a-video.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/gen-1.png",
      "bytes": 787896,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-249c9ee11fa47dfeba95.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/gen-1.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/video-LDM.png",
      "bytes": 275362,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-8e093390771a3e4196fa.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/video-LDM.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/video-LDM-autoencoder.png",
      "bytes": 2216305,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-82178fac6e387b9b4e47.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/video-LDM-autoencoder.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/lumiere.png",
      "bytes": 238170,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-fc9b9b7c2b0d529b940e.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/lumiere.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/lumiere-STUnet.png",
      "bytes": 777041,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-bad122fc4bb86dfff262.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/lumiere-STUnet.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/text2video-zero.png",
      "bytes": 1336632,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-7f69c80e2beb3b08d8bd.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/text2video-zero.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/control-video.png",
      "bytes": 283837,
      "reused": false,
      "local_path": "attachments/38eb95b0e9e6/images/body-d325ac8e1fbb25ceedf1.png",
      "src": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/control-video.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://lilianweng.github.io/posts/2021-07-11-diffusion-models/",
      "label": "Diffusion models",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2010.02502",
      "label": "Song et al. 2020",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2202.00512",
      "label": "Salimans & Ho (2022)",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2204.03458",
      "label": "Ho & Salimans, et al. 2022",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1606.06650",
      "label": "Cicek et al. 2016",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v2/",
      "label": "relative position",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2210.02303",
      "label": "Ho, et al. 2022",
      "type": "a"
    },
    {
      "href": "https://lilianweng.github.io/posts/2019-01-31-lm/",
      "label": "T5",
      "type": "a"
    },
    {
      "href": "https://chao-ji.github.io/jekyll/update/2018/07/19/BilinearResize.html",
      "label": "bilinear resizing",
      "type": "a"
    },
    {
      "href": "https://openai.com/research/video-generation-models-as-world-simulators",
      "label": "Brooks et al. 2024",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2209.14792",
      "label": "Singer et al. 2022",
      "type": "a"
    },
    {
      "href": "https://openaccess.thecvf.com/content/ICCV2023/html/Wu_Tune-A-Video_One-Shot_Tuning_of_Image_Diffusion_Models_for_Text-to-Video_Generation_ICCV_2023_paper.html",
      "label": "Wu et al. 2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2302.03011",
      "label": "Esser et al. 2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2304.08818",
      "label": "Blattmann et al. 2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2311.15127",
      "label": "Blattmann et al. 2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2401.12945",
      "label": "Bar-Tal et al. 2024",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2303.13439",
      "label": "Khachatryan et al. 2023",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/2305.13077",
      "label": "Zhang et al. 2023",
      "type": "a"
    },
    {
      "href": "https://video-diffusion.github.io/",
      "label": "webpage",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:51:12Z",
  "source_url": "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2010.02502",
      "bytes": 10857218,
      "reused": false,
      "local_path": "papers/repair/2e617588af84373a622f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2202.00512",
      "bytes": 3488589,
      "reused": false,
      "local_path": "papers/repair/353b5d2204f89a0a3e98.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2204.03458",
      "bytes": 8008990,
      "reused": false,
      "local_path": "papers/repair/4d330c49d1cea8c13aaf.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1606.06650",
      "bytes": 625976,
      "reused": false,
      "local_path": "papers/repair/b7ce2d79976fb429d263.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2210.02303",
      "bytes": 8893839,
      "reused": false,
      "local_path": "papers/repair/96d235f2b174708eabfb.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2209.14792",
      "bytes": 14464869,
      "reused": false,
      "local_path": "papers/repair/3d61a63fe37463590a3d.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2302.03011",
      "bytes": 48060418,
      "reused": false,
      "local_path": "papers/repair/b1722bdff4b5a323c35f.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2304.08818",
      "bytes": 12061068,
      "reused": false,
      "local_path": "papers/repair/9e7c2267c2bc294db8ba.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2311.15127",
      "bytes": 15373867,
      "reused": false,
      "local_path": "papers/repair/3fbc8a7a76754af54861.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2401.12945",
      "bytes": 26789812,
      "reused": false,
      "local_path": "papers/repair/392f64b7e29839e83fb7.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2303.13439",
      "bytes": 22261780,
      "reused": false,
      "local_path": "papers/repair/9588d1065a518bc9ce4a.pdf"
    },
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/2305.13077",
      "bytes": 14985026,
      "reused": false,
      "local_path": "papers/repair/2ab0ba3a3d0a793ce671.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 30630,
  "expected_body_images": 15,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/38eb95b0e9e6/pages/source.html",
  "local_html": "attachments/38eb95b0e9e6/pages/article.html",
  "body_base_path": "attachments/38eb95b0e9e6/pages/",
  "body_markdown_path": "attachments/38eb95b0e9e6/pages/article.md"
}