{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "zeromathai",
  "title_listed": "Approximate Inference: Sampling the Posterior vs Optimizing an Approximation",
  "keywords": "",
  "domain": "大模型算法",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/zeromathai/approximate-inference-sampling-the-posterior-vs-optimizing-an-approximation-1nbo",
  "publish_date": "2026-09-10",
  "corpus_stage": "",
  "notes": "",
  "id": "ed793939c85b",
  "collected_at": "2026-10-02T13:50:30",
  "title": "Approximate Inference: Sampling the Posterior vs Optimizing an Approximation - DEV Community",
  "body_markdown": "In a simple probabilistic model, posterior inference sounds straightforward: observe some data, compute the posterior distribution over hidden variables, and use it to answer the question you care about.\nIn complex models, that computation is often the hard part. The latent space may be high-dimensional, the posterior may have a complicated shape, and expectations under that posterior may be difficult to compute analytically.\nThe practical question becomes:\nIf the exact posterior is difficult to compute, how can we turn inference into something tractable?\nApproximate Inference gives us two major answers:\nMCMC:\napproach the posterior through sampling.\nVariational Inference:\napproximate the posterior with a distribution found through optimization.\nBoth address the same posterior-inference problem, but they make it computationally manageable in different ways.\nThe core mental model\nSuppose\nx\nx\nx\nis observed data and\nz\nz\nz\nrepresents hidden variables. Posterior inference is concerned with\np\n(\nz\n∣\nx\n)\np(z\\mid x)\np\n(\nz\n∣\nx\n)\nand, when needed, expectations under that distribution.\nThe problem is not simply defining the posterior. It is computing with it.\nWhen multiple hidden variables interact, their joint states determine the shape of the posterior. As those interactions become more complex, directly representing the exact posterior or calculating expectations under it becomes increasingly difficult.\nApproximate inference changes the computational problem instead of insisting on an exact closed-form calculation.\nExact posterior is difficult to compute\n                |\n        Approximate Inference\n          /             \\\n         /               \\\n      MCMC                 VI\n       |                   |\n   Sampling           Optimization\n       |                   |\nMarkov Chain       Variational Family\n       |                   |\nPosterior-like      Best approximate\nsamples             distribution\nEnter fullscreen mode\nExit fullscreen mode\nThe key distinction is simple: MCMC approaches the posterior through samples, while VI searches for a tractable distribution that can stand in for it.\nMCMC: represent the posterior with samples\nMarkov Chain Monte Carlo starts from the fact that directly obtaining the samples we want from a complicated posterior may be difficult.\nInstead of calculating the posterior itself in closed form, MCMC constructs a stochastic process that approaches that posterior over time.\nThe goal is to build an\nergodic Markov Chain\nover the latent-variable space whose\nStationary Distribution\nis the target posterior.\nThe computational flow is:\nConstruct an ergodic Markov Chain\nwhose stationary distribution\nis the target posterior\n                |\n                v\nRun the chain\n                |\n                v\nCollect states as samples\n                |\n                v\nEstimate posterior quantities empirically\nEnter fullscreen mode\nExit fullscreen mode\nErgodicity matters because the chain should not remain trapped indefinitely in a single starting state. As it progresses, the chain must be able to explore the state space needed to approach its target Stationary Distribution.\nOnce the chain has approached that distribution, collected states can be used to approximate properties of the posterior. Metropolis-Hastings and Gibbs Sampling are representative examples of this MCMC approach.\nThe useful implementation-level mental model is that\nthe samples become the practical representation of the posterior\n. You do not need a convenient closed-form expression for the entire distribution if you can obtain samples that reflect it well enough to estimate the quantities you need.\nThat flexibility comes with computational cost. MCMC may require many repeated transitions, and it can be difficult to determine whether the collected samples are sufficiently independent. For complex models or large problems, repeated sampling can therefore become expensive.\nThis also shows where approximation enters MCMC. The method does not approximate the target posterior by choosing a simpler distribution family. With sufficient computation, it can approach the exact result.\nIn practice, however, processor time is finite, so only a limited number of samples can be collected.\nFor MCMC, finite computation is the practical source of approximation.\nWhy high-dimensional sampling is difficult\nThe role of the Markov Chain becomes easier to understand when we look at sampling in high-dimensional spaces.\nNot every part of a large state space matters equally. The full space may be enormous, while useful samples can be concentrated in regions to which the target distribution assigns high probability.\nRandomly choosing points from the entire space is therefore an inefficient way to find the regions that matter.\nMCMC approaches this problem sequentially. Starting from some state, the chain repeatedly moves to another state. It first needs to reach a High-Probability Region and can then continue exploring regions with substantial probability under the target distribution.\nThe mental model is not:\nSearch the entire high-dimensional space at once\nEnter fullscreen mode\nExit fullscreen mode\nIt is closer to:\ncurrent state\n     |\n     v\nnext state\n     |\n     v\nnext state\n     |\n     v\nhigh-probability region\n     |\n     v\ncontinue exploring and sampling\nEnter fullscreen mode\nExit fullscreen mode\nRandom-walk behavior provides useful intuition here. The point is not that MCMC learns a separate manifold.\nThe important idea is that\nfinding probability-concentrated regions is itself difficult in high dimensions, and a Markov Chain provides a way to move toward and explore those regions\n.\nThat is why chain construction and sample collection are central to the method.\nVariational Inference: turn inference into optimization\nVariational Inference takes the same posterior-inference problem and makes it tractable in a different way.\nInstead of repeatedly collecting samples that reflect the exact posterior, VI approximates the difficult posterior with a probability distribution that is easier to work with.\nThis turns inference into an optimization problem.\nThe first step is to define a\nVariational Family\nQ\n\\mathcal{Q}\nQ\n, a set of candidate distributions that are allowed to approximate the exact posterior.\nEach\nq\n(\nz\n)\n∈\nQ\nq(z)\\in\\mathcal{Q}\nq\n(\nz\n)\n∈\nQ\nis a candidate approximation to\np\n(\nz\n∣\nx\n)\np(z\\mid x)\np\n(\nz\n∣\nx\n)\n. To keep the approximation tractable, the family may use a Factorized Approximation or a Parametric Form such as a Gaussian.\nThe next step is to select the candidate that is closest to the exact posterior.\nVI expresses this as:\nq\n∗\n(\nz\n)\n=\nargmin\n⁡\nq\n(\nz\n)\n∈\nQ\nD\nK\nL\n(\nq\n(\nz\n)\n∣\np\n(\nz\n∣\nx\n)\n)\nq^*(z)=\\underset{q(z)\\in\\mathcal{Q}}{\\operatorname{argmin}}\\;D_{\\mathrm{KL}}\\left(q(z)\\,|\\,p(z\\mid x)\\right)\nq\n∗\n(\nz\n)\n=\nq\n(\nz\n)\n∈\nQ\nargmin\n​\nD\nKL\n​\n(\nq\n(\nz\n)\n∣\np\n(\nz\n∣\nx\n)\n)\nHere,\nQ\n\\mathcal{Q}\nQ\nis the Variational Family,\nq\n(\nz\n)\nq(z)\nq\n(\nz\n)\nis a candidate approximate distribution,\np\n(\nz\n∣\nx\n)\np(z\\mid x)\np\n(\nz\n∣\nx\n)\nis the Exact Posterior, and\nq\n∗\n(\nz\n)\nq^*(z)\nq\n∗\n(\nz\n)\nis the optimal approximation within\nQ\n\\mathcal{Q}\nQ\naccording to KL Divergence.\nThe KL Divergence being minimized is:\nD\nK\nL\n(\nq\n(\nz\n)\n∣\np\n(\nz\n∣\nx\n)\n)\n=\nE\nz\n∼\nq\n(\nz\n)\n[\nlog\n⁡\nq\n(\nz\n)\np\n(\nz\n∣\nx\n)\n]\nD_{\\mathrm{KL}}\\left(q(z)\\,|\\,p(z\\mid x)\\right)=\\mathbb{E}_{z\\sim q(z)}\\left[\\log\\frac{q(z)}{p(z\\mid x)}\\right]\nD\nKL\n​\n(\nq\n(\nz\n)\n∣\np\n(\nz\n∣\nx\n)\n)\n=\nE\nz\n∼\nq\n(\nz\n)\n​\n[\nlo\ng\np\n(\nz\n∣\nx\n)\nq\n(\nz\n)\n​\n]\nThe two equations play different roles.\nThe first defines the\noptimization problem\n: choose the candidate distribution in\nQ\n\\mathcal{Q}\nQ\nthat minimizes the divergence from the exact posterior.\nThe second defines the\nquantity being minimized\n: the KL Divergence expressed as an expectation under\nq\n(\nz\n)\nq(z)\nq\n(\nz\n)\n.\nThe overall calculation flow is:\nStart with the difficult Exact Posterior\np\n(\nz\n∣\nx\n)\np(z\\mid x)\np\n(\nz\n∣\nx\n)\n.\nChoose a Variational Family\nQ\n\\mathcal{Q}\nQ\n.\nConsider candidate distributions\nq\n(\nz\n)\n∈\nQ\nq(z)\\in\\mathcal{Q}\nq\n(\nz\n)\n∈\nQ\n.\nMeasure their KL Divergence from\np\n(\nz\n∣\nx\n)\np(z\\mid x)\np\n(\nz\n∣\nx\n)\n.\nOptimize within\nQ\n\\mathcal{Q}\nQ\n.\nObtain the optimal approximation\nq\n∗\n(\nz\n)\nq^*(z)\nq\n∗\n(\nz\n)\n.\nUse\nq\n∗\n(\nz\n)\nq^*(z)\nq\n∗\n(\nz\n)\nin place of the Exact Posterior for subsequent inference calculations.\nThis is the core idea behind\nInference as Optimization\n.\nMCMC explores the posterior through a sequence of samples. VI searches through a restricted space of probability distributions.\nWhere VI's approximation comes from\nRestricting the search space is what makes VI easier to compute, but that restriction also creates its main limitation.\nSuppose the true posterior cannot be represented well by the chosen Variational Family\nQ\n\\mathcal{Q}\nQ\n. Optimization can still find the best possible\nq\n∗\n(\nz\n)\nq^*(z)\nq\n∗\n(\nz\n)\nwithin that family, but even a perfectly solved optimization problem cannot produce a distribution outside the family that was allowed from the start.\nThis is where VI differs fundamentally from MCMC.\nFor VI, the approximation is not simply:\nWe stopped computing too early.\nEnter fullscreen mode\nExit fullscreen mode\nIt is:\nWe deliberately restricted\nwhat the approximate posterior\nis allowed to look like.\nEnter fullscreen mode\nExit fullscreen mode\nFactorized or parametric assumptions can make inference tractable, but they also limit which posterior shapes can be represented.\nFor VI, the restricted analytical approximation is itself the source of approximation.\nMCMC vs VI: the distinction that matters\nBoth methods address the same problem: posterior inference when direct exact computation is difficult.\nTheir computational structures, however, are different.\nMCMC\nVariational Inference\nCore approach\nSampling\nOptimization\nPosterior representation\nCollected samples\nApproximate distribution\nq\n(\nz\n)\nq(z)\nq\n(\nz\n)\nMain structure\nMarkov Chain\nVariational Family\nQ\n\\mathcal{Q}\nQ\nTarget\nStationary Distribution is the posterior\nMinimize KL Divergence to the posterior\nPractical source of approximation\nFinite computation\nRestricted approximation family\nComputational characteristic\nRepeated sampling can be costly\nSome VI methods can scale relatively well to large applications\nWhen reading a probabilistic model or inference implementation, the most useful question is:\nWhere has the original posterior-inference problem been moved?\nWith MCMC, it has been moved into the behavior of a Markov Chain. You construct the chain so that its Stationary Distribution is the target posterior, run it, collect samples, and estimate posterior quantities empirically.\nWith VI, it has been moved into an optimization problem. You define\nQ\n\\mathcal{Q}\nQ\n, search within that family, and use\nq\n∗\n(\nz\n)\nq^*(z)\nq\n∗\n(\nz\n)\nas the tractable replacement for\np\n(\nz\n∣\nx\n)\np(z\\mid x)\np\n(\nz\n∣\nx\n)\n.\nThat structural difference is the most important mental model to carry into equations, models, and implementations.\nTakeaway\nApproximate Inference is a way to make difficult Posterior Inference computationally manageable.\nMCMC says:\nDo not compute the entire posterior directly. Construct a Markov Chain whose Stationary Distribution is the posterior, then use samples from that chain.\nVariational Inference says:\nDo not compute the difficult posterior directly. Search for a tractable distribution that approximates it.\nThe crucial difference is where approximation enters the computation. In MCMC, practical approximation comes from finite sampling and finite computation. In VI, approximation comes from restricting the posterior to a chosen Variational Family.\nBoth are computational answers to the same question:\nwhat should we do when the posterior we want is too difficult to compute exactly?\nOriginally published at zeromathai.com.\nOriginal article:\nhttps://zeromathai.com/en/approximate-inference-course-en/",
  "metadata": {
    "title": "Approximate Inference: Sampling the Posterior vs Optimizing an Approximation - DEV Community",
    "description": "In a simple probabilistic model, posterior inference sounds straightforward: observe some data,... Tagged with machinelearning, bayesian, mcmc, variationalinference.",
    "keywords": "machinelearning, bayesian, mcmc, variationalinference, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/zeromathai/approximate-inference-sampling-the-posterior-vs-optimizing-an-approximation-1nbo",
    "og:title": "Approximate Inference: Sampling the Posterior vs Optimizing an Approximation",
    "og:description": "In a simple probabilistic model, posterior inference sounds straightforward: observe some data,...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpinkgzfefk7vr112k1d2.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Approximate Inference: Sampling the Posterior vs Optimizing an Approximation",
    "twitter:description": "In a simple probabilistic model, posterior inference sounds straightforward: observe some data,...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpinkgzfefk7vr112k1d2.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/ed793939c85b/code/body-000.txt",
      "chars": 401,
      "language": "text"
    },
    {
      "file": "attachments/ed793939c85b/code/body-001.txt",
      "chars": 278,
      "language": "text"
    },
    {
      "file": "attachments/ed793939c85b/code/body-002.txt",
      "chars": 49,
      "language": "text"
    },
    {
      "file": "attachments/ed793939c85b/code/body-003.txt",
      "chars": 148,
      "language": "text"
    },
    {
      "file": "attachments/ed793939c85b/code/body-004.txt",
      "chars": 32,
      "language": "text"
    },
    {
      "file": "attachments/ed793939c85b/code/body-005.txt",
      "chars": 83,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "https://zeromathai.com/en/approximate-inference-course-en/",
      "label": "https://zeromathai.com/en/approximate-inference-course-en/",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:14Z",
  "source_url": "https://dev.to/zeromathai/approximate-inference-sampling-the-posterior-vs-optimizing-an-approximation-1nbo",
  "papers": [],
  "actual_video_files": [],
  "body_chars": 11792,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "ok",
  "missing_reason": "",
  "raw_html": "attachments/ed793939c85b/pages/source.html",
  "local_html": "attachments/ed793939c85b/pages/article.html",
  "body_base_path": "attachments/ed793939c85b/pages/",
  "body_markdown_path": "attachments/ed793939c85b/pages/article.md"
}