{ "source_file": "0930", "source_platform": "个人博客", "author": "Lilian Weng / Lil'Log", "title_listed": "From GAN to WGAN", "keywords": "GAN,WGAN,生成模型", "domain": "机器学习", "content_type": "深度技术解析", "is_paid_flag": "否", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/", "publish_date": "2017-08", "corpus_stage": "预训练/SFT", "notes": "GAN到WGAN的演进综述", "id": "45444954e783", "collected_at": "2026-10-02T14:13:27", "title": "From GAN to WGAN | Lil'Log", "body_markdown": "[Updated on 2018-09-30: thanks to Yoonju, we have this post translated in\nKorean\n!]\n[Updated on 2019-04-18: this post is also available on\narXiv\n.]\nGenerative adversarial network\n(GAN) has shown great results in many generative tasks to replicate the real-world rich content such as images, human language, and music. It is inspired by game theory: two models, a generator and a critic, are competing with each other while making each other stronger at the same time. However, it is rather challenging to train a GAN model, as people are facing issues like training instability or failure to converge.\nHere I would like to explain the maths behind the generative adversarial network framework, why it is hard to be trained, and finally introduce a modified version of GAN intended to solve the training difficulties.\nKullback–Leibler and Jensen–Shannon Divergence\n#\nBefore we start examining GANs closely, let us first review two metrics for quantifying the similarity between two probability distributions.\n(1)\nKL (Kullback–Leibler) divergence\nmeasures how one probability distribution $p$ diverges from a second expected probability distribution $q$.\n$$\nD_{KL}(p \\| q) = \\int_x p(x) \\log \\frac{p(x)}{q(x)} dx\n$$\n$D_{KL}$ achieves the minimum zero when $p(x)$ == $q(x)$ everywhere.\nIt is noticeable according to the formula that KL divergence is asymmetric. In cases where $p(x)$ is close to zero, but $q(x)$ is significantly non-zero, the $q$’s effect is disregarded. It could cause buggy results when we just want to measure the similarity between two equally important distributions.\n(2)\nJensen–Shannon Divergence\nis another measure of similarity between two probability distributions, bounded by $[0, 1]$. JS divergence is symmetric (yay!) and more smooth. Check this\nQuora post\nif you are interested in reading more about the comparison between KL divergence and JS divergence.\n$$\nD_{JS}(p \\| q) = \\frac{1}{2} D_{KL}(p \\| \\frac{p + q}{2}) + \\frac{1}{2} D_{KL}(q \\| \\frac{p + q}{2})\n$$\nGiven two Gaussian distribution, $p$ with mean=0 and std=1 and $q$ with mean=1 and std=1. The average of two distributions is labelled as $m=(p+q)/2$. KL divergence $D_{KL}$ is asymmetric but JS divergence $D_{JS}$ is symmetric.\nSome believe (\nHuszar, 2015\n) that one reason behind GANs’ big success is switching the loss function from asymmetric KL divergence in traditional maximum-likelihood approach to symmetric JS divergence. We will discuss more on this point in the next section.\nGenerative Adversarial Network (GAN)\n#\nGAN consists of two models:\nA discriminator $D$ estimates the probability of a given sample coming from the real dataset. It works as a critic and is optimized to tell the fake samples from the real ones.\nA generator $G$ outputs synthetic samples given a noise variable input $z$ ($z$ brings in potential output diversity). It is trained to capture the real data distribution so that its generative samples can be as real as possible, or in other words, can trick the discriminator to offer a high probability.\nArchitecture of a generative adversarial network. (Image source:\nwww.kdnuggets.com/2017/01/generative-...-learning.html\n)\nThese two models compete against each other during the training process: the generator $G$ is trying hard to trick the discriminator, while the critic model $D$ is trying hard not to be cheated. This interesting zero-sum game between two models motivates both to improve their functionalities.\nGiven,\nSymbol\nMeaning\nNotes\n$p_{z}$\nData distribution over noise input $z$\nUsually, just uniform.\n$p_{g}$\nThe generator’s distribution over data $x$\n$p_{r}$\nData distribution over real sample $x$\nOn one hand, we want to make sure the discriminator $D$’s decisions over real data are accurate by maximizing $\\mathbb{E}_{x \\sim p_{r}(x)} [\\log D(x)]$. Meanwhile, given a fake sample $G(z), z \\sim p_z(z)$, the discriminator is expected to output a probability, $D(G(z))$, close to zero by maximizing $\\mathbb{E}_{z \\sim p_{z}(z)} [\\log (1 - D(G(z)))]$.\nOn the other hand, the generator is trained to increase the chances of $D$ producing a high probability for a fake example, thus to minimize $\\mathbb{E}_{z \\sim p_{z}(z)} [\\log (1 - D(G(z)))]$.\nWhen combining both aspects together, $D$ and $G$ are playing a\nminimax game\nin which we should optimize the following loss function:\n$$\n\\begin{aligned}\n\\min_G \\max_D L(D, G) \n& = \\mathbb{E}_{x \\sim p_{r}(x)} [\\log D(x)] + \\mathbb{E}_{z \\sim p_z(z)} [\\log(1 - D(G(z)))] \\\\\n& = \\mathbb{E}_{x \\sim p_{r}(x)} [\\log D(x)] + \\mathbb{E}_{x \\sim p_g(x)} [\\log(1 - D(x)]\n\\end{aligned}\n$$\n($\\mathbb{E}_{x \\sim p_{r}(x)} [\\log D(x)]$ has no impact on $G$ during gradient descent updates.)\nWhat is the optimal value for D?\n#\nNow we have a well-defined loss function. Let’s first examine what is the best value for $D$.\n$$\nL(G, D) = \\int_x \\bigg( p_{r}(x) \\log(D(x)) + p_g (x) \\log(1 - D(x)) \\bigg) dx\n$$\nSince we are interested in what is the best value of $D(x)$ to maximize $L(G, D)$, let us label\n$$\n\\tilde{x} = D(x), \nA=p_{r}(x), \nB=p_g(x)\n$$\nAnd then what is inside the integral (we can safely ignore the integral because $x$ is sampled over all the possible values) is:\n$$\n\\begin{aligned}\nf(\\tilde{x}) \n& = A log\\tilde{x} + B log(1-\\tilde{x}) \\\\\n\\frac{d f(\\tilde{x})}{d \\tilde{x}}\n& = A \\frac{1}{ln10} \\frac{1}{\\tilde{x}} - B \\frac{1}{ln10} \\frac{1}{1 - \\tilde{x}} \\\\\n& = \\frac{1}{ln10} (\\frac{A}{\\tilde{x}} - \\frac{B}{1-\\tilde{x}}) \\\\\n& = \\frac{1}{ln10} \\frac{A - (A + B)\\tilde{x}}{\\tilde{x} (1 - \\tilde{x})} \\\\\n\\end{aligned}\n$$\nThus, set $\\frac{d f(\\tilde{x})}{d \\tilde{x}} = 0$, we get the best value of the discriminator: $D^*(x) = \\tilde{x}^* = \\frac{A}{A + B} = \\frac{p_{r}(x)}{p_{r}(x) + p_g(x)} \\in [0, 1]$.\nOnce the generator is trained to its optimal, $p_g$ gets very close to $p_{r}$. When $p_g = p_{r}$, $D^*(x)$ becomes $1/2$.\nWhat is the global optimal?\n#\nWhen both $G$ and $D$ are at their optimal values, we have $p_g = p_{r}$ and $D^*(x) = 1/2$ and the loss function becomes:\n$$\n\\begin{aligned}\nL(G, D^*) \n&= \\int_x \\bigg( p_{r}(x) \\log(D^*(x)) + p_g (x) \\log(1 - D^*(x)) \\bigg) dx \\\\\n&= \\log \\frac{1}{2} \\int_x p_{r}(x) dx + \\log \\frac{1}{2} \\int_x p_g(x) dx \\\\\n&= -2\\log2\n\\end{aligned}\n$$\nWhat does the loss function represent?\n#\nAccording to the formula listed in the\nprevious section\n, JS divergence between $p_{r}$ and $p_g$ can be computed as:\n$$\n\\begin{aligned}\nD_{JS}(p_{r} \\| p_g) \n=& \\frac{1}{2} D_{KL}(p_{r} || \\frac{p_{r} + p_g}{2}) + \\frac{1}{2} D_{KL}(p_{g} || \\frac{p_{r} + p_g}{2}) \\\\\n=& \\frac{1}{2} \\bigg( \\log2 + \\int_x p_{r}(x) \\log \\frac{p_{r}(x)}{p_{r} + p_g(x)} dx \\bigg) + \\\\& \\frac{1}{2} \\bigg( \\log2 + \\int_x p_g(x) \\log \\frac{p_g(x)}{p_{r} + p_g(x)} dx \\bigg) \\\\\n=& \\frac{1}{2} \\bigg( \\log4 + L(G, D^*) \\bigg)\n\\end{aligned}\n$$\nThus,\n$$\nL(G, D^*) = 2D_{JS}(p_{r} \\| p_g) - 2\\log2\n$$\nEssentially the loss function of GAN quantifies the similarity between the generative data distribution $p_g$ and the real sample distribution $p_{r}$ by JS divergence when the discriminator is optimal. The best $G^*$ that replicates the real data distribution leads to the minimum $L(G^*, D^*) = -2\\log2$ which is aligned with equations above.\nOther Variations of GAN\n: There are many variations of GANs in different contexts or designed for different tasks. For example, for semi-supervised learning, one idea is to update the discriminator to output real class labels, $1, \\dots, K-1$, as well as one fake class label $K$. The generator model aims to trick the discriminator to output a classification label smaller than $K$.\nTensorflow Implementation\n:\ncarpedm20/DCGAN-tensorflow\nProblems in GANs\n#\nAlthough GAN has shown great success in the realistic image generation, the training is not easy; The process is known to be slow and unstable.\nHard to achieve Nash equilibrium\n#\nSalimans et al. (2016)\ndiscussed the problem with GAN’s gradient-descent-based training procedure. Two models are trained simultaneously to find a\nNash equilibrium\nto a two-player non-cooperative game. However, each model updates its cost independently with no respect to another player in the game. Updating the gradient of both models concurrently cannot guarantee a convergence.\nLet’s check out a simple example to better understand why it is difficult to find a Nash equilibrium in an non-cooperative game. Suppose one player takes control of $x$ to minimize $f_1(x) = xy$, while at the same time the other player constantly updates $y$ to minimize $f_2(y) = -xy$.\nBecause $\\frac{\\partial f_1}{\\partial x} = y$ and $\\frac{\\partial f_2}{\\partial y} = -x$, we update $x$ with $x-\\eta \\cdot y$ and $y$ with $y+ \\eta \\cdot x$ simulitanously in one iteration, where $\\eta$ is the learning rate. Once $x$ and $y$ have different signs, every following gradient update causes huge oscillation and the instability gets worse in time, as shown in\nA simulation of our example for updating $x$ to minimize $xy$ and updating $y$ to minimize $-xy$. The learning rate $\\eta = 0.1$. With more iterations, the oscillation grows more and more unstable.\nLow dimensional supports\n#\nTerm\nExplanation\nManifold\nA topological space that locally resembles Euclidean space near each point. Precisely, when this Euclidean space is of\ndimension $n$\n, the manifold is referred as\n$n$-manifold\n.\nSupport\nA real-valued function $f$ is the subset of the domain containing those elements which are not mapped to\nzero\n.\nArjovsky and Bottou (2017)\ndiscussed the problem of the\nsupports\nof $p_r$ and $p_g$ lying on low dimensional\nmanifolds\nand how it contributes to the instability of GAN training thoroughly in a very theoretical paper\n“Towards principled methods for training generative adversarial networks”\n.\nThe dimensions of many real-world datasets, as represented by $p_r$, only appear to be\nartificially high\n. They have been found to concentrate in a lower dimensional manifold. This is actually the fundamental assumption for\nManifold Learning\n. Thinking of the real world images, once the theme or the contained object is fixed, the images have a lot of restrictions to follow, i.e., a dog should have two ears and a tail, and a skyscraper should have a straight and tall body, etc. These restrictions keep images aways from the possibility of having a high-dimensional free form.\n$p_g$ lies in a low dimensional manifolds, too. Whenever the generator is asked to a much larger image like 64x64 given a small dimension, such as 100, noise variable input $z$, the distribution of colors over these 4096 pixels has been defined by the small 100-dimension random number vector and can hardly fill up the whole high dimensional space.\nBecause both $p_g$ and $p_r$ rest in low dimensional manifolds, they are almost certainly gonna be disjoint (See Fig. 4). When they have disjoint supports, we are always capable of finding a perfect discriminator that separates real and fake samples 100% correctly. Check the\npaper\nif you are curious about the proof.\nLow dimensional manifolds in high dimension space can hardly have overlaps. (Left) Two lines in a three-dimension space. (Right) Two surfaces in a three-dimension space.\nVanishing gradient\n#\nWhen the discriminator is perfect, we are guaranteed with $D(x) = 1, \\forall x \\in p_r$ and $D(x) = 0, \\forall x \\in p_g$. Therefore the loss function $L$ falls to zero and we end up with no gradient to update the loss during learning iterations. Fig. 5 demonstrates an experiment when the discriminator gets better, the gradient vanishes fast.\nFirst, a DCGAN is trained for 1, 10 and 25 epochs. Then, with the **generator fixed**, a discriminator is trained from scratch and measure the gradients with the original cost function. We see the gradient norms **decay quickly** (in log scale), in the best case 5 orders of magnitude after 4000 discriminator iterations. (Image source:\nArjovsky and Bottou, 2017\n)\nAs a result, training a GAN faces a\ndilemma\n:\nIf the discriminator behaves badly, the generator does not have accurate feedback and the loss function cannot represent the reality.\nIf the discriminator does a great job, the gradient of the loss function drops down to close to zero and the learning becomes super slow or even jammed.\nThis dilemma clearly is capable to make the GAN training very tough.\nMode collapse\n#\nDuring the training, the generator may collapse to a setting where it always produces same outputs. This is a common failure case for GANs, commonly referred to as\nMode Collapse\n. Even though the generator might be able to trick the corresponding discriminator, it fails to learn to represent the complex real-world data distribution and gets stuck in a small space with extremely low variety.\nA DCGAN model is trained with an MLP network with 4 layers, 512 units and ReLU activation function, configured to lack a strong inductive bias for image generation. The results shows a significant degree of mode collapse. (Image source:\nArjovsky, Chintala, & Bottou, 2017.\n)\nLack of a proper evaluation metric\n#\nGenerative adversarial networks are not born with a good objection function that can inform us the training progress. Without a good evaluation metric, it is like working in the dark. No good sign to tell when to stop; No good indicator to compare the performance of multiple models.\nImproved GAN Training\n#\nThe following suggestions are proposed to help stabilize and improve the training of GANs.\nFirst five methods are practical techniques to achieve faster convergence of GAN training, proposed in\n“Improve Techniques for Training GANs”\n.\nThe last two are proposed in\n“Towards principled methods for training generative adversarial networks”\nto solve the problem of disjoint distributions.\n(1)\nFeature Matching\nFeature matching suggests to optimize the discriminator to inspect whether the generator’s output matches expected statistics of the real samples. In such a scenario, the new loss function is defined as $| \\mathbb{E}_{x \\sim p_r} f(x) - \\mathbb{E}_{z \\sim p_z(z)}f(G(z)) |_2^2 $, where $f(x)$ can be any computation of statistics of features, such as mean or median.\n(2)\nMinibatch Discrimination\nWith minibatch discrimination, the discriminator is able to digest the relationship between training data points in one batch, instead of processing each point independently.\nIn one minibatch, we approximate the closeness between every pair of samples, $c(x_i, x_j)$, and get the overall summary of one data point by summing up how close it is to other samples in the same batch, $o(x_i) = \\sum_{j} c(x_i, x_j)$. Then $o(x_i)$ is explicitly added to the input of the model.\n(3)\nHistorical Averaging\nFor both models, add $ | \\Theta - \\frac{1}{t} \\sum_{i=1}^t \\Theta_i |^2 $ into the loss function, where $\\Theta$ is the model parameter and $\\Theta_i$ is how the parameter is configured at the past training time $i$. This addition piece penalizes the training speed when $\\Theta$ is changing too dramatically in time.\n(4)\nOne-sided Label Smoothing\nWhen feeding the discriminator, instead of providing 1 and 0 labels, use soften values such as 0.9 and 0.1. It is shown to reduce the networks’ vulnerability.\n(5)\nVirtual Batch Normalization\n(VBN)\nEach data sample is normalized based on a fixed batch (\n“reference batch”\n) of data rather than within its minibatch. The reference batch is chosen once at the beginning and stays the same through the training.\nTheano Implementation\n:\nopenai/improved-gan\n(6)\nAdding Noises\n.\nBased on the discussion in the\nprevious section\n, we now know $p_r$ and $p_g$ are disjoint in a high dimensional space and it causes the problem of vanishing gradient. To artificially “spread out” the distribution and to create higher chances for two probability distributions to have overlaps, one solution is to add continuous noises onto the inputs of the discriminator $D$.\n(7)\nUse Better Metric of Distribution Similarity\nThe loss function of the vanilla GAN measures the JS divergence between the distributions of $p_r$ and $p_g$. This metric fails to provide a meaningful value when two distributions are disjoint.\nWasserstein metric\nis proposed to replace JS divergence because it has a much smoother value space. See more in the next section.\nWasserstein GAN (WGAN)\n#\nWhat is Wasserstein distance?\n#\nWasserstein Distance\nis a measure of the distance between two probability distributions.\nIt is also called\nEarth Mover’s distance\n, short for EM distance, because informally it can be interpreted as the minimum energy cost of moving and transforming a pile of dirt in the shape of one probability distribution to the shape of the other distribution. The cost is quantified by: the amount of dirt moved x the moving distance.\nLet us first look at a simple case where the probability domain is\ndiscrete\n. For example, suppose we have two distributions $P$ and $Q$, each has four piles of dirt and both have ten shovelfuls of dirt in total. The numbers of shovelfuls in each dirt pile are assigned as follows:\n$$\n\\begin{aligned}\n& P_1 = 3, P_2 = 2, P_3 = 1, P_4 = 4\\\\\n& Q_1 = 1, Q_2 = 2, Q_3 = 4, Q_4 = 3\n\\end{aligned}\n$$\nIn order to change $P$ to look like $Q$, as illustrated in Fig. 7, we:\nFirst move 2 shovelfuls from $P_1$ to $P_2$ => $(P_1, Q_1)$ match up.\nThen move 2 shovelfuls from $P_2$ to $P_3$ => $(P_2, Q_2)$ match up.\nFinally move 1 shovelfuls from $Q_3$ to $Q_4$ => $(P_3, Q_3)$ and $(P_4, Q_4)$ match up.\nIf we label the cost to pay to make $P_i$ and $Q_i$ match as $\\delta_i$, we would have $\\delta_{i+1} = \\delta_i + P_i - Q_i$ and in the example:\n$$\n\\begin{aligned}\n\\delta_0 &= 0\\\\\n\\delta_1 &= 0 + 3 - 1 = 2\\\\\n\\delta_2 &= 2 + 2 - 2 = 2\\\\\n\\delta_3 &= 2 + 1 - 4 = -1\\\\\n\\delta_4 &= -1 + 4 - 3 = 0\n\\end{aligned}\n$$\nFinally the Earth Mover’s distance is $W = \\sum \\vert \\delta_i \\vert = 5$.\nStep-by-step plan of moving dirt between piles in $P$ and $Q$ to make them match.\nWhen dealing with the continuous probability domain, the distance formula becomes:\n$$\nW(p_r, p_g) = \\inf_{\\gamma \\sim \\Pi(p_r, p_g)} \\mathbb{E}_{(x, y) \\sim \\gamma}[\\| x-y \\|]\n$$\nIn the formula above, $\\Pi(p_r, p_g)$ is the set of all possible joint probability distributions between $p_r$ and $p_g$. One joint distribution $\\gamma \\in \\Pi(p_r, p_g)$ describes one dirt transport plan, same as the discrete example above, but in the continuous probability space. Precisely $\\gamma(x, y)$ states the percentage of dirt should be transported from point $x$ to $y$ so as to make $x$ follows the same probability distribution of $y$. That’s why the marginal distribution over $x$ adds up to $p_g$, $\\sum_{x} \\gamma(x, y) = p_g(y)$ (Once we finish moving the planned amount of dirt from every possible $x$ to the target $y$, we end up with exactly what $y$ has according to $p_g$.) and vice versa $\\sum_{y} \\gamma(x, y) = p_r(x)$.\nWhen treating $x$ as the starting point and $y$ as the destination, the total amount of dirt moved is $\\gamma(x, y)$ and the travelling distance is $| x-y |$ and thus the cost is $\\gamma(x, y) \\cdot | x-y |$. The expected cost averaged across all the $(x,y)$ pairs can be easily computed as:\n$$\n\\sum_{x, y} \\gamma(x, y) \\| x-y \\| \n= \\mathbb{E}_{x, y \\sim \\gamma} \\| x-y \\|\n$$\nFinally, we take the minimum one among the costs of all dirt moving solutions as the EM distance. In the definition of Wasserstein distance, the $\\inf$ (\ninfimum\n, also known as\ngreatest lower bound\n) indicates that we are only interested in the smallest cost.\nWhy Wasserstein is better than JS or KL divergence?\n#\nEven when two distributions are located in lower dimensional manifolds without overlaps, Wasserstein distance can still provide a meaningful and smooth representation of the distance in-between.\nThe WGAN paper exemplified the idea with a simple example.\nSuppose we have two probability distributions, $P$ and $Q$:\n$$\n\\forall (x, y) \\in P, x = 0 \\text{ and } y \\sim U(0, 1)\\\\\n\\forall (x, y) \\in Q, x = \\theta, 0 \\leq \\theta \\leq 1 \\text{ and } y \\sim U(0, 1)\\\\\n$$\nThere is no overlap between $P$ and $Q$ when $\\theta \\neq 0$.\nWhen $\\theta \\neq 0$:\n$$\n\\begin{aligned}\nD_{KL}(P \\| Q) &= \\sum_{x=0, y \\sim U(0, 1)} 1 \\cdot \\log\\frac{1}{0} = +\\infty \\\\\nD_{KL}(Q \\| P) &= \\sum_{x=\\theta, y \\sim U(0, 1)} 1 \\cdot \\log\\frac{1}{0} = +\\infty \\\\\nD_{JS}(P, Q) &= \\frac{1}{2}(\\sum_{x=0, y \\sim U(0, 1)} 1 \\cdot \\log\\frac{1}{1/2} + \\sum_{x=0, y \\sim U(0, 1)} 1 \\cdot \\log\\frac{1}{1/2}) = \\log 2\\\\\nW(P, Q) &= |\\theta|\n\\end{aligned}\n$$\nBut when $\\theta = 0$, two distributions are fully overlapped:\n$$\n\\begin{aligned}\nD_{KL}(P \\| Q) &= D_{KL}(Q \\| P) = D_{JS}(P, Q) = 0\\\\\nW(P, Q) &= 0 = \\lvert \\theta \\rvert\n\\end{aligned}\n$$\n$D_{KL}$ gives us inifity when two distributions are disjoint. The value of $D_{JS}$ has sudden jump, not differentiable at $\\theta = 0$. Only Wasserstein metric provides a smooth measure, which is super helpful for a stable learning process using gradient descents.\nUse Wasserstein distance as GAN loss function\n#\nIt is intractable to exhaust all the possible joint distributions in $\\Pi(p_r, p_g)$ to compute $\\inf_{\\gamma \\sim \\Pi(p_r, p_g)}$. Thus the authors proposed a smart transformation of the formula based on the Kantorovich-Rubinstein duality to:\n$$\nW(p_r, p_g) = \\frac{1}{K} \\sup_{\\| f \\|_L \\leq K} \\mathbb{E}_{x \\sim p_r}[f(x)] - \\mathbb{E}_{x \\sim p_g}[f(x)]\n$$\nwhere $\\sup$ (\nsupremum\n) is the opposite of $inf$ (infimum); we want to measure the least upper bound or, in even simpler words, the maximum value.\nLipschitz continuity?\nThe function $f$ in the new form of Wasserstein metric is demanded to satisfy $| f |_L \\leq K$, meaning it should be\nK-Lipschitz continuous\n.\nA real-valued function $f: \\mathbb{R} \\rightarrow \\mathbb{R}$ is called $K$-Lipschitz continuous if there exists a real constant $K \\geq 0$ such that, for all $x_1, x_2 \\in \\mathbb{R}$,\n$$\n\\lvert f(x_1) - f(x_2) \\rvert \\leq K \\lvert x_1 - x_2 \\rvert\n$$\nHere $K$ is known as a Lipschitz constant for function $f(.)$. Functions that are everywhere continuously differentiable is Lipschitz continuous, because the derivative, estimated as $\\frac{\\lvert f(x_1) - f(x_2) \\rvert}{\\lvert x_1 - x_2 \\rvert}$, has bounds. However, a Lipschitz continuous function may not be everywhere differentiable, such as $f(x) = \\lvert x \\rvert$.\nExplaining how the transformation happens on the Wasserstein distance formula is worthy of a long post by itself, so I skip the details here. If you are interested in how to compute Wasserstein metric using linear programming, or how to transfer Wasserstein metric into its dual form according to the Kantorovich-Rubinstein Duality, read this\nawesome post\n.\nSuppose this function $f$ comes from a family of K-Lipschitz continuous functions, $\\{ f_w \\}_{w \\in W}$, parameterized by $w$. In the modified Wasserstein-GAN, the “discriminator” model is used to learn $w$ to find a good $f_w$ and the loss function is configured as measuring the Wasserstein distance between $p_r$ and $p_g$.\n$$\nL(p_r, p_g) = W(p_r, p_g) = \\max_{w \\in W} \\mathbb{E}_{x \\sim p_r}[f_w(x)] - \\mathbb{E}_{z \\sim p_r(z)}[f_w(g_\\theta(z))]\n$$\nThus the “discriminator” is not a direct critic of telling the fake samples apart from the real ones anymore. Instead, it is trained to learn a $K$-Lipschitz continuous function to help compute Wasserstein distance. As the loss function decreases in the training, the Wasserstein distance gets smaller and the generator model’s output grows closer to the real data distribution.\nOne big problem is to maintain the $K$-Lipschitz continuity of $f_w$ during the training in order to make everything work out. The paper presents a simple but very practical trick: After every gradient update, clamp the weights $w$ to a small window, such as $[-0.01, 0.01]$, resulting in a compact parameter space $W$ and thus $f_w$ obtains its lower and upper bounds to preserve the Lipschitz continuity.\nAlgorithm of Wasserstein generative adversarial network. (Image source:\nArjovsky, Chintala, & Bottou, 2017.\n)\nCompared to the original GAN algorithm, the WGAN undertakes the following changes:\nAfter every gradient update on the critic function, clamp the weights to a small fixed range, $[-c, c]$.\nUse a new loss function derived from the Wasserstein distance, no logarithm anymore. The “discriminator” model does not play as a direct critic but a helper for estimating the Wasserstein metric between real and generated data distribution.\nEmpirically the authors recommended\nRMSProp\noptimizer on the critic, rather than a momentum based optimizer such as\nAdam\nwhich could cause instability in the model training. I haven’t seen clear theoretical explanation on this point through.\nSadly, Wasserstein GAN is not perfect. Even the authors of the original WGAN paper mentioned that\n“Weight clipping is a clearly terrible way to enforce a Lipschitz constraint”\n(Oops!). WGAN still suffers from unstable training, slow convergence after weight clipping (when clipping window is too large), and vanishing gradients (when clipping window is too small).\nSome improvement, precisely replacing weight clipping with\ngradient penalty\n, has been discussed in\nGulrajani et al. 2017\n. I will leave this to a future post.\nExample: Create New Pokemons!\n#\nJust for fun, I tried out\ncarpedm20/DCGAN-tensorflow\non a tiny dataset,\nPokemon sprites\n. The dataset only has 900-ish pokemon images, including different levels of same pokemon species.\nLet’s check out what types of new pokemons the model is able to create.\nUnfortunately due to the tiny training data, the new pokemons only have rough shapes without details. The shapes and colors do look better with more training epoches! Hooray!\nTrain\ncarpedm20/DCGAN-tensorflow\non a set of Pokemon sprite images. The sample outputs are listed after training epoches = 7, 21, 49.\nIf you are interested in a commented version of\ncarpedm20/DCGAN-tensorflow\nand how to modify it to train WGAN and WGAN with gradient penalty, check\nlilianweng/unified-gan-tensorflow\n.\nCited as:\n@article{weng2017gan,\n title = \"From GAN to WGAN\",\n author = \"Weng, Lilian\",\n journal = \"lilianweng.github.io\",\n year = \"2017\",\n url = \"https://lilianweng.github.io/posts/2017-08-20-gan/\"\n}\nOR\n@misc{weng2019gan,\n title={From GAN to WGAN},\n author={Lilian Weng},\n year={2019},\n eprint={1904.08994},\n archivePrefix={arXiv},\n primaryClass={cs.LG}\n}\nReferences\n#\n[1] Goodfellow, Ian, et al.\n“Generative adversarial nets.”\nNIPS, 2014.\n[2] Tim Salimans, et al.\n“Improved techniques for training gans.”\nNIPS 2016.\n[3] Martin Arjovsky and Léon Bottou.\n“Towards principled methods for training generative adversarial networks.”\narXiv preprint arXiv:1701.04862 (2017).\n[4] Martin Arjovsky, Soumith Chintala, and Léon Bottou.\n“Wasserstein GAN.”\narXiv preprint arXiv:1701.07875 (2017).\n[5] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron Courville.\nImproved training of wasserstein gans.\narXiv preprint arXiv:1704.00028 (2017).\n[6]\nComputing the Earth Mover’s Distance under Transformations\n[7]\nWasserstein GAN and the Kantorovich-Rubinstein Duality\n[8]\nzhuanlan.zhihu.com/p/25071913\n[9] Ferenc Huszár.\n“How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?.”\narXiv preprint arXiv:1511.05101 (2015).", "metadata": { "title": "From GAN to WGAN | Lil'Log", "description": "[Updated on 2018-09-30: thanks to Yoonju, we have this post translated in Korean!]\n\n[Updated on 2019-04-18: this post is also available on arXiv.]\nGenerative adversarial network (GAN) has shown great results in many generative tasks to replicate the real-world rich content such as images, human language, and music. It is inspired by game theory: two models, a generator and a critic, are competing with each other while making each other stronger at the same time. However, it is rather challenging to train a GAN model, as people are facing issues like training instability or failure to converge.", "keywords": "gan, long-read, generative-model, image-generation, math-heavy", "author": "Lilian Weng", "og:title": "From GAN to WGAN", "og:description": "[Updated on 2018-09-30: thanks to Yoonju, we have this post translated in Korean!]\n\n[Updated on 2019-04-18: this post is also available on arXiv.]\nGenerative adversarial network (GAN) has shown great results in many generative tasks to replicate the real-world rich content such as images, human language, and music. It is inspired by game theory: two models, a generator and a critic, are competing with each other while making each other stronger at the same time. However, it is rather challenging to train a GAN model, as people are facing issues like training instability or failure to converge.", "og:type": "article", "og:url": "https://lilianweng.github.io/posts/2017-08-20-gan/", "twitter:card": "summary", "twitter:title": "From GAN to WGAN", "twitter:description": "[Updated on 2018-09-30: thanks to Yoonju, we have this post translated in Korean!]\n\n[Updated on 2019-04-18: this post is also available on arXiv.]\nGenerative adversarial network (GAN) has shown great results in many generative tasks to replicate the real-world rich content such as images, human language, and music. It is inspired by game theory: two models, a generator and a critic, are competing with each other while making each other stronger at the same time. However, it is rather challenging to train a GAN model, as people are facing issues like training instability or failure to converge.", "article:section": "posts", "article:published_time": "2017-08-20T00:00:00+00:00", "article:modified_time": "2017-08-20T00:00:00+00:00" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [ { "file": "attachments/45444954e783/code/body-000.txt", "chars": 205, "language": "text" }, { "file": "attachments/45444954e783/code/body-001.txt", "chars": 171, "language": "text" } ], "images": [ { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/KL_JS_divergence.png", "bytes": 221077, "reused": false, "local_path": "attachments/45444954e783/images/body-aa28a4cc1b3516c1be46.png", "src": "https://lilianweng.github.io/posts/2017-08-20-gan/KL_JS_divergence.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/GAN.png", "bytes": 157401, "reused": false, "local_path": "attachments/45444954e783/images/body-4199969ec35e7360ae98.png", "src": "https://lilianweng.github.io/posts/2017-08-20-gan/GAN.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/nash_equilibrium.png", "bytes": 101972, "reused": false, "local_path": "attachments/45444954e783/images/body-95c24c614030b59aff01.png", "src": "https://lilianweng.github.io/posts/2017-08-20-gan/nash_equilibrium.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/low_dim_manifold.png", "bytes": 447186, "reused": false, "local_path": "attachments/45444954e783/images/body-c2ebec0990753f6d5a87.png", "src": "https://lilianweng.github.io/posts/2017-08-20-gan/low_dim_manifold.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/GAN_vanishing_gradient.png", "bytes": 736874, "reused": false, "local_path": "attachments/45444954e783/images/body-f32623608491cb056216.png", "src": "https://lilianweng.github.io/posts/2017-08-20-gan/GAN_vanishing_gradient.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/mode_collapse.png", "bytes": 529052, "reused": false, "local_path": "attachments/45444954e783/images/body-54028801a0ddfc354538.png", "src": "https://lilianweng.github.io/posts/2017-08-20-gan/mode_collapse.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/EM_distance_discrete.png", "bytes": 86758, "reused": false, "local_path": "attachments/45444954e783/images/body-501aa5d6dea1582e9385.png", "src": "https://lilianweng.github.io/posts/2017-08-20-gan/EM_distance_discrete.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/wasserstein_simple_example.png", "bytes": 32150, "reused": false, "local_path": "attachments/45444954e783/images/body-7301ab34ef06aa0e487d.png", "src": "https://lilianweng.github.io/posts/2017-08-20-gan/wasserstein_simple_example.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/WGAN_algorithm.png", "bytes": 313647, "reused": false, "local_path": "attachments/45444954e783/images/body-bacb4aa725276c994c68.png", "src": "https://lilianweng.github.io/posts/2017-08-20-gan/WGAN_algorithm.png", "alt": "", "downloaded": true }, { "status": "ok", "url": "https://lilianweng.github.io/posts/2017-08-20-gan/pokemon-GAN.png", "bytes": 3106088, "reused": false, "local_path": "attachments/45444954e783/images/body-f670e3a31b7940e1d8ef.png", "src": "https://lilianweng.github.io/posts/2017-08-20-gan/pokemon-GAN.png", "alt": "", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://github.com/yjucho1/articles/blob/master/fromGANtoWGAN/readme.md", "label": "Korean", "type": "a" }, { "href": "https://arxiv.org/abs/1904.08994", "label": "arXiv", "type": "a" }, { "href": "https://arxiv.org/pdf/1406.2661.pdf", "label": "Generative adversarial network", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence", "label": "KL (Kullback–Leibler) divergence", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence", "label": "Jensen–Shannon Divergence", "type": "a" }, { "href": "https://www.quora.com/Why-isnt-the-Jensen-Shannon-divergence-used-more-often-than-the-Kullback-Leibler-since-JS-is-symmetric-thus-possibly-a-better-indicator-of-distance", "label": "Quora post", "type": "a" }, { "href": "https://arxiv.org/pdf/1511.05101.pdf", "label": "Huszar, 2015", "type": "a" }, { "href": "http://www.kdnuggets.com/2017/01/generative-adversarial-networks-hot-topic-machine-learning.html", "label": "www.kdnuggets.com/2017/01/generative-...-learning.html", "type": "a" }, { "href": "https://lilianweng.github.io/posts/2017-08-20-gan/", "label": "previous section", "type": "a" }, { "href": "https://github.com/carpedm20/DCGAN-tensorflow", "label": "carpedm20/DCGAN-tensorflow", "type": "a" }, { "href": "http://papers.nips.cc/paper/6125-improved-techniques-for-training-gans.pdf", "label": "Salimans et al. (2016)", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Nash_equilibrium", "label": "Nash equilibrium", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Manifold", "label": "Manifold", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Support_(mathematics)", "label": "Support", "type": "a" }, { "href": "https://arxiv.org/pdf/1701.04862.pdf", "label": "Arjovsky and Bottou (2017)", "type": "a" }, { "href": "http://scikit-learn.org/stable/modules/manifold.html", "label": "Manifold Learning", "type": "a" }, { "href": "https://arxiv.org/pdf/1701.07875.pdf", "label": "Arjovsky, Chintala, & Bottou, 2017.", "type": "a" }, { "href": "https://github.com/openai/improved-gan", "label": "openai/improved-gan", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Wasserstein_metric", "label": "Wasserstein metric", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Infimum_and_supremum", "label": "infimum", "type": "a" }, { "href": "https://en.wikipedia.org/wiki/Lipschitz_continuity", "label": "K-Lipschitz continuous", "type": "a" }, { "href": "https://vincentherrmann.github.io/blog/wasserstein/", "label": "awesome post", "type": "a" }, { "href": "http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf", "label": "RMSProp", "type": "a" }, { "href": "https://arxiv.org/abs/1412.6980v8", "label": "Adam", "type": "a" }, { "href": "https://arxiv.org/pdf/1704.00028.pdf", "label": "Gulrajani et al. 2017", "type": "a" }, { "href": "https://github.com/PokeAPI/sprites/", "label": "Pokemon sprites", "type": "a" }, { "href": "https://github.com/lilianweng/unified-gan-tensorflow", "label": "lilianweng/unified-gan-tensorflow", "type": "a" }, { "href": "http://robotics.stanford.edu/~scohen/research/emdg/emdg.html", "label": "Computing the Earth Mover’s Distance under Transformations", "type": "a" }, { "href": "https://zhuanlan.zhihu.com/p/25071913", "label": "zhuanlan.zhihu.com/p/25071913", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:51:34Z", "source_url": "https://lilianweng.github.io/posts/2017-08-20-gan/", "papers": [ { "status": "ok", "url": "https://arxiv.org/pdf/1904.08994", "bytes": 2372969, "reused": false, "local_path": "papers/repair/a08caad93b55901c5b6f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1406.2661.pdf", "bytes": 530482, "reused": false, "local_path": "papers/repair/ee65f498ce3f3e038de2.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1511.05101.pdf", "bytes": 505788, "reused": false, "local_path": "papers/repair/057d39e637c6097bcd7a.pdf" }, { "status": "ok", "url": "http://papers.nips.cc/paper/6125-improved-techniques-for-training-gans.pdf", "bytes": 2639054, "reused": false, "local_path": "papers/repair/762153e376c22f644d34.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1701.04862.pdf", "bytes": 566179, "reused": false, "local_path": "papers/repair/e9832d130ee863bb324f.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1701.07875.pdf", "bytes": 8834587, "reused": false, "local_path": "papers/repair/941e1ba609c7c19a8af8.pdf" }, { "status": "ok", "url": "http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf", "bytes": 546843, "reused": false, "local_path": "papers/repair/7b52598737bfafc8deab.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1412.6980v8", "bytes": 619544, "reused": false, "local_path": "papers/repair/0a612e9d0294ad8485d1.pdf" }, { "status": "ok", "url": "https://arxiv.org/pdf/1704.00028.pdf", "bytes": 6187582, "reused": false, "local_path": "papers/repair/e6aa0075133ef9410890.pdf" } ], "actual_video_files": [], "body_chars": 27088, "expected_body_images": 10, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/45444954e783/pages/source.html", "local_html": "attachments/45444954e783/pages/article.html", "body_base_path": "attachments/45444954e783/pages/", "body_markdown_path": "attachments/45444954e783/pages/article.md" }