{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "zeromathai",
  "title_listed": "Deep Generative Models: Four Ways to Make Complex Distributions Learnable",
  "keywords": "",
  "domain": "文本处理",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/zeromathai/deep-generative-models-four-ways-to-make-complex-distributions-learnable-o43",
  "publish_date": "2026-09-17",
  "corpus_stage": "",
  "notes": "",
  "id": "bd353e461ae9",
  "collected_at": "2026-10-02T13:50:26",
  "title": "Deep Generative Models: Four Ways to Make Complex Distributions Learnable - DEV Community",
  "body_markdown": "When developers first encounter deep generative models, it is easy to reduce them to one idea:\nmodels that generate new data\n.\nThat is only the visible outcome. The deeper problem is deciding\nhow to represent a complex data distribution in a form that a model can compute and learn\n.\nHigh-dimensional data such as images and audio can follow extremely complex distributions. A single simple probability model is not enough to represent them directly, so different generative model families restructure the problem in different ways.\nAn Autoregressive Model factorizes a joint distribution into conditional distributions. A Variational Autoencoder (VAE) introduces latent variables. A Flow-based Model starts from a simple distribution and transforms it through invertible mappings. A Generative Adversarial Network (GAN) takes a different route and learns a generation process without directly evaluating an explicit probability density.\nA useful mental model is:\nDeep generative modeling is not just about producing samples. It is about turning a difficult distribution-modeling problem into something the model can compute and learn.\nTwo Broad Families\nOne useful distinction is between\nLikelihood-based\nand\nLikelihood-Free\napproaches.\nLikelihood-based models include:\nAutoregressive Models\nVariational Autoencoders\nFlow-based Models\nDiffusion Models\nScore-based Models\nGAN is a representative Likelihood-Free model.\nThis distinction matters because it reveals the structural choice behind each model family. A complex distribution can be decomposed into conditional distributions, represented indirectly through latent variables, transformed from a simpler distribution, or learned without directly computing its probability density.\nDiffusion Models and Score-based Models also belong to the Likelihood-based side of this classification. Their detailed mechanisms, including Forward Diffusion, Reverse Diffusion, and Score Matching, are outside the scope here.\nFrom KL Divergence to NLL\nFor Likelihood-based modeling, a natural goal is to make the model distribution close to the data distribution.\nLet the true data distribution be\np\nd\na\nt\na\np_{\\mathrm{data}}\np\ndata\n​\nand the model distribution parameterized by\nθ\n\\theta\nθ\nbe\np\nm\no\nd\ne\nl\np_{\\mathrm{model}}\np\nmodel\n​\n.\nWe can express that goal by minimizing KL Divergence:\nmin\n⁡\nθ\nD\nK\nL\n[\np\nd\na\nt\na\n(\nx\n)\n∣\np\nm\no\nd\ne\nl\n(\nx\n;\nθ\n)\n]\n\\min_{\\theta} D_{\\mathrm{KL}}\\left[p_{\\mathrm{data}}(x)\\,|\\,p_{\\mathrm{model}}(x;\\theta)\\right]\nθ\nmin\n​\nD\nKL\n​\n[\np\ndata\n​\n(\nx\n)\n∣\np\nmodel\n​\n(\nx\n;\nθ\n)\n]\nExpanding the KL Divergence gives:\nD\nK\nL\n[\np\nd\na\nt\na\n∣\np\nm\no\nd\ne\nl\n]\nE\nx\n∼\np\nd\na\nt\na\n[\nlog\n⁡\np\nd\na\nt\na\n(\nx\n)\n]\nE\nx\n∼\np\nd\na\nt\na\n[\nlog\n⁡\np\nm\no\nd\ne\nl\n(\nx\n;\nθ\n)\n]\nD_{\\mathrm{KL}}\\left[p_{\\mathrm{data}}\\,|\\,p_{\\mathrm{model}}\\right]\n\n\n  \n  \n  \\mathbb{E}{x\\sim p{\\mathrm{data}}}\\left[\\log p_{\\mathrm{data}}(x)\\right]\n\n\n\\mathbb{E}{x\\sim p{\\mathrm{data}}}\\left[\\log p_{\\mathrm{model}}(x;\\theta)\\right]\nD\nKL\n​\n[\np\ndata\n​\n∣\np\nmodel\n​\n]\nE\nx\n∼\np\ndata\n[\nlo\ng\np\ndata\n​\n(\nx\n)\n]\nE\nx\n∼\np\ndata\n[\nlo\ng\np\nmodel\n​\n(\nx\n;\nθ\n)\n]\nThe first term depends only on the data distribution. If\np\nd\na\nt\na\np_{\\mathrm{data}}\np\ndata\n​\nis fixed, changing\nθ\n\\theta\nθ\ndoes not affect it.\nThe model-dependent part is therefore:\nmin\n⁡\nθ\nE\nx\n∼\np\nd\na\nt\na\n[\n−\nlog\n⁡\np\nm\no\nd\ne\nl\n(\nx\n;\nθ\n)\n]\n\\min_{\\theta}\n\\mathbb{E}{x\\sim p{\\mathrm{data}}}\n\\left[-\\log p_{\\mathrm{model}}(x;\\theta)\\right]\nθ\nmin\n​\nE\nx\n∼\np\ndata\n[\n−\nlo\ng\np\nmodel\n​\n(\nx\n;\nθ\n)\n]\nThis is the Cross-Entropy term between the data distribution and the model distribution.\nIn practice, we do not know the full underlying data distribution, so the expectation is approximated using training samples. The resulting objective is\nNegative Log-Likelihood (NLL) minimization\n.\nThe optimization flow is:\nMinimize KL Divergence\n        ↓\nRemove the entropy term independent of θ\n        ↓\nMinimize Cross-Entropy\n        ↓\nMinimize NLL on training data\nEnter fullscreen mode\nExit fullscreen mode\nThe intuition is simple: assign high probability to regions where real data occurs.\nThis connection is the basic picture when the model provides a way to work explicitly with Likelihood. When\np\n(\nx\n)\np(x)\np\n(\nx\n)\nbecomes difficult to optimize directly, the model needs a different structure.\nThat is where the major generative model families begin to diverge.\nStrategy 1: Factorize the Distribution\nAn Autoregressive Model starts with a straightforward idea: if the full joint distribution is difficult to model at once, break it into a sequence of conditional distributions.\nThe Probability Chain Rule gives an exact factorization:\np\n(\nx\n)\n=\n∏\ni\n=\n1\nn\np\n(\nx\ni\n∣\nx\n1\n,\n…\n,\nx\ni\n−\n1\n)\np(x)=\\prod_{i=1}^{n}p(x_i\\mid x_1,\\ldots,x_{i-1})\np\n(\nx\n)\n=\ni\n=\n1\n∏\nn\n​\np\n(\nx\ni\n​\n∣\nx\n1\n​\n,\n…\n,\nx\ni\n−\n1\n​\n)\nThis is not an approximation introduced for convenience. It is an exact decomposition of the joint distribution.\nThe calculation proceeds through conditional terms such as\np\n(\nx\n1\n)\np(x_1)\np\n(\nx\n1\n​\n)\n, then\np\n(\nx\n2\n∣\nx\n1\n)\np(x_2\\mid x_1)\np\n(\nx\n2\n​\n∣\nx\n1\n​\n)\n, then\np\n(\nx\n3\n∣\nx\n1\n,\nx\n2\n)\np(x_3\\mid x_1,x_2)\np\n(\nx\n3\n​\n∣\nx\n1\n​\n,\nx\n2\n​\n)\n, continuing until\np\n(\nx\nn\n∣\nx\n1\n,\n…\n,\nx\nn\n−\n1\n)\np(x_n\\mid x_1,\\ldots,x_{n-1})\np\n(\nx\nn\n​\n∣\nx\n1\n​\n,\n…\n,\nx\nn\n−\n1\n​\n)\n.\nThat changes the modeling problem. Instead of asking how to represent one complicated\np\n(\nx\n)\np(x)\np\n(\nx\n)\ndirectly, we ask how well the network can model each conditional distribution.\nGPT, RNN, PixelRNN, PixelCNN, WaveNet, NADE, and MADE use different data and network structures, but they share this conditional factorization principle.\nFrom an implementation perspective, the important thing to inspect is the\nordering and conditioning structure\n. Each prediction must depend on the variables that precede it in the chosen factorization.\nStrategy 2: Introduce a Latent Variable\nA VAE restructures the problem differently.\nInstead of factorizing the observed variables sequentially, it introduces an unobserved latent variable\nz\nz\nz\nand models the observed data\nx\nx\nx\nthrough that latent state.\nThe data distribution is written as:\np\n(\nx\n)\n=\n∫\np\n(\nz\n)\np\n(\nx\n∣\nz\n)\nd\nz\np(x)=\\int p(z)p(x\\mid z)\\,dz\np\n(\nx\n)\n=\n∫\np\n(\nz\n)\np\n(\nx\n∣\nz\n)\nd\nz\nHere,\nz\nz\nz\nis the latent variable,\np\n(\nz\n)\np(z)\np\n(\nz\n)\ndefines its distribution, and\np\n(\nx\n∣\nz\n)\np(x\\mid z)\np\n(\nx\n∣\nz\n)\ndescribes how an observation is generated from a particular latent state.\nThe structural flow is:\nLatent state\n     ↓\nConditional generation\n     ↓\nObserved data\nEnter fullscreen mode\nExit fullscreen mode\nIn probabilistic terms, the Decoder models\np\n(\nx\n∣\nz\n)\np(x\\mid z)\np\n(\nx\n∣\nz\n)\n.\nThe VAE also needs a way to infer which latent states are plausible for a given observation. Its\nEncoder\n, also called the Recognition Network, does not simply map\nx\nx\nx\nto one fixed latent value. It constructs an approximate distribution over\nz\nz\nz\ngiven\nx\nx\nx\n.\nThe\nDecoder\n, or Generative Network, models the conditional distribution of\nx\nx\nx\ngiven\nz\nz\nz\n.\nThe difficulty is that the exact Posterior over\nz\nz\nz\ngiven an observation can be hard to compute. VAE addresses this with\nVariational Inference\nand trains by maximizing an\nELBO\n.\nThe important point here is not the full ELBO derivation. It is the structural move: VAE combines latent-variable modeling with approximate inference so that a difficult\np\n(\nx\n)\np(x)\np\n(\nx\n)\nbecomes a learnable problem.\nStrategy 3: Transform a Simple Distribution\nNormalizing Flow takes another approach.\nWhere an Autoregressive Model decomposes a complex\np\n(\nx\n)\np(x)\np\n(\nx\n)\ninto conditional distributions, a Flow-based Model begins with a simple distribution and changes its shape through a sequence of transformations.\nThe structure is:\nSimple Distribution\n        ↓\nInvertible Transformation\n        ↓\nInvertible Transformation\n        ↓\n        ...\n        ↓\nTarget Distribution\nEnter fullscreen mode\nExit fullscreen mode\nThe key requirement is\ninvertibility\n.\nIt is not enough to map values from a simple distribution into a more complicated one. The transformation must also allow the model to trace a transformed value back to where it came from.\nThat property keeps changes in probability density computable through the sequence of transformations.\nThis requirement also constrains the model architecture. A Flow-based Model cannot freely use arbitrary transformations if they violate invertibility.\nThe benefit is that the final data variable\nx\nx\nx\nhas an\nexplicit probability density\n. Its Likelihood can therefore be computed, and training can directly optimize NLL on the observed data.\nA useful implementation-level mental model is:\nPreserve invertibility so that probability density remains computable while the distribution becomes progressively more complex.\nSaying that a Flow-based Model models the data distribution does not mean the learned distribution becomes identical to the true data distribution. It means the architecture defines an explicit model distribution\np\n(\nx\n)\np(x)\np\n(\nx\n)\nwhose Likelihood can be computed and optimized.\nStrategy 4: Learn Without Direct Density Evaluation\nGAN changes the setup more substantially.\nInstead of putting the explicit value of\np\n(\nx\n)\np(x)\np\n(\nx\n)\nat the center of training, GAN introduces two competing models:\na\nGenerator\n, which produces samples resembling real data,\na\nDiscriminator\n, which tries to distinguish real samples from generated ones.\nTheir opposing objectives form a\nMinimax Problem\n.\nThe structural view is:\nGenerator\n    ↓\nGenerated samples\n    ↓\nDiscriminator\n ↙           ↘\nReal      Generated\ndata        data\nEnter fullscreen mode\nExit fullscreen mode\nAs the Generator produces more convincing samples, the Discriminator has a harder time separating generated data from real data. As the Discriminator improves, the Generator is pushed to produce better samples.\nThe important difference from an Autoregressive Model or a Flow-based Model is that GAN training does\nnot\ncenter on directly computing an explicit\np\n(\nx\n)\np(x)\np\n(\nx\n)\nfor each observation.\nInstead, the interaction between the Generator and Discriminator provides the learning signal that pushes the Generator toward producing samples similar to those from the real data distribution.\nThat is why GAN is treated as a Likelihood-Free approach in this comparison.\nThe Structural Difference That Matters\nThe most useful way to compare these models is not by memorizing architecture names, but by asking how each one turns a complex distribution into a problem the model can compute and learn.\nModel\nStructural move\nCore training view\nAutoregressive Model\nFactorize\np\n(\nx\n)\np(x)\np\n(\nx\n)\ninto conditional distributions\nModel each conditional probability\nVAE\nIntroduce latent variable\nz\nz\nz\nApproximate inference and optimize ELBO\nFlow-based Model\nApply invertible transformations\nExplicit density and NLL\nGAN\nAvoid direct explicit density evaluation\nGenerator–Discriminator Minimax learning\nThese approaches begin with the same fundamental difficulty: complex high-dimensional data distributions are hard to model directly.\nWhat changes is the structure used to make learning possible.\nTakeaway\nWhen reading the implementation of a generative model, do not start with the layer names.\nStart with the probability structure. Check whether the model factorizes the distribution, introduces latent variables and approximate inference, preserves density through invertible transformations, or learns through an adversarial objective without directly evaluating an explicit density.\nOnce that structural choice is clear, the architecture, objective, and training procedure become much easier to understand as parts of the same generative-modeling problem.\nOriginally published at zeromathai.com.\nOriginal article:\nhttps://zeromathai.com/en/deep-generative-models-course-en/",
  "metadata": {
    "title": "Deep Generative Models: Four Ways to Make Complex Distributions Learnable - DEV Community",
    "description": "When developers first encounter deep generative models, it is easy to reduce them to one idea: models... Tagged with machinelearning, generativeai, deeplearning, probability.",
    "keywords": "machinelearning, generativeai, deeplearning, probability, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/zeromathai/deep-generative-models-four-ways-to-make-complex-distributions-learnable-o43",
    "og:title": "Deep Generative Models: Four Ways to Make Complex Distributions Learnable",
    "og:description": "When developers first encounter deep generative models, it is easy to reduce them to one idea: models...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1welm8okfndlw6l5zo89.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Deep Generative Models: Four Ways to Make Complex Distributions Learnable",
    "twitter:description": "When developers first encounter deep generative models, it is easy to reduce them to one idea: models...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1welm8okfndlw6l5zo89.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/bd353e461ae9/code/body-000.txt",
      "chars": 147,
      "language": "text"
    },
    {
      "file": "attachments/bd353e461ae9/code/body-001.txt",
      "chars": 64,
      "language": "text"
    },
    {
      "file": "attachments/bd353e461ae9/code/body-002.txt",
      "chars": 144,
      "language": "text"
    },
    {
      "file": "attachments/bd353e461ae9/code/body-003.txt",
      "chars": 106,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://zeromathai.com/en/deep-generative-models-course-en/",
      "label": "https://zeromathai.com/en/deep-generative-models-course-en/",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:13Z",
  "source_url": "https://dev.to/zeromathai/deep-generative-models-four-ways-to-make-complex-distributions-learnable-o43",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11677,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/bd353e461ae9/pages/source.html",
  "local_html": "attachments/bd353e461ae9/pages/article.html",
  "body_base_path": "attachments/bd353e461ae9/pages/",
  "body_markdown_path": "attachments/bd353e461ae9/pages/article.md"
}