{ "source_file": "0930", "source_platform": "Dev.to", "author": "Mitansh Gor", "title_listed": "GEN-AI 5 : WGAN and WGAN-GP", "keywords": "", "domain": "图像处理", "content_type": "技术教程", "is_paid_flag": "否", "url": "https://dev.to/mitanshgor/gen-ai-2-generative-ai-and-auto-encoders-do8", "publish_date": "2025-04-20", "corpus_stage": "", "notes": "", "id": "a627609d3fba", "collected_at": "2026-10-02T13:53:13", "title": "GEN-AI 5 : WGAN and WGAN-GP - DEV Community", "body_markdown": "Why ❌ GAN\nIn the ever-evolving landscape of generative models, GANs have taken center stage with their remarkable ability to generate data that mimics real-world distributions. But as with all great things, classic GANs came with caveats—training instability, vanishing gradients, and mode collapse, to name a few.\nLet’s dive deeper into these powerful models: WGAN and its enhanced cousin WGAN-GP, two sophisticated upgrades that fix many of GANs' shortcomings.\nWhat is Wasserstein distance\nWasserstein Distance, also known as the Earth Mover's Distance (EMD), is a mathematical measure of the distance between two probability distributions.\nImagine you have two piles of sand—one symbolizing real data and the other representing data generated by a model. The goal is to reshape one pile into the other by moving portions of sand. The effort required depends not only on how much sand you move but also on how far you move it. This effort is what the Wasserstein Distance measures—a cost-efficient way to transform one distribution into another.\nWhat makes this distance especially powerful is its ability to compare distributions even when they don't overlap at all—a situation where traditional measures like JS divergence fail. It offers a continuous and interpretable signal for how \"far off\" the generated data is from the real data, making it incredibly useful for training generative models like WGANs.\nTraditional GANs use Jensen-Shannon (JS) divergence to measure similarity between distributions.\nBUT :\nJS divergence becomes undefined or uninformative when distributions don't overlap.\nThis leads to vanishing gradients—a huge problem for training\nBy contrast, Wasserstein Distance:\nRemains well-defined and continuous even when distributions are far apart.\nProvides meaningful gradients, allowing the generator to learn effectively from the start.\nImprovement in WGAN wrt GAN\nWGAN is a variant of GAN that uses the Wasserstein Distance as its loss function instead of JS divergence.\nKey differences from traditional GAN:\nThe discriminator is now called a critic. It doesn't classify data as real/fake, but scores it to reflect how “real” it looks.\nThe loss function is based on the difference in critic scores for real vs. fake data.\nIn simple terms:\nReal Data → High Critic Score\nFake Data → Low Critic Score\nThe generator improves by producing data that the critic scores higher.\nWorking\nJust like in standard GANs, WGAN consists of two neural networks:\nA Generator (G) that tries to generate realistic data from random noise.\nA Critic (C) (not called a discriminator here) that scores the realness of data, assigning higher values to real data and lower values to fake ones.\nUnlike GANs, which use a\nsigmoid function\nto classify samples as real or fake, WGAN's critic outputs real-valued scores—this small shift changes everything.\nThe Loss Function\nThe heart of WGAN lies in replacing Jensen-Shannon divergence with the Wasserstein distance (also called Earth Mover’s Distance)—a metric that measures how much \"effort\" it takes to morph the generated distribution into the real one.\nThis results in the following loss functions:\nCritic's Loss:\n𝐿c = E[C(fake)]−E[C(real)]\nThe critic aims to maximize the difference, rewarding real data with higher scores and penalizing fakes.\nGenerator's Loss:\n𝐿G = −E[C(fake)]\nThe generator tries to minimize the critic’s ability to detect its fakes by generating better, more realistic data.\nMissing Something? 🤔\nThe model that we learned until now has some issues.\nThe critic assigns real-valued scores to data samples, so nothing stops the critic from amplifying its outputs to maximize the loss. For example, it might assign real samples a score of +10,000. And fake samples a score of -10,000. This leads to :\nThe generator receiving extreme gradients, which are not helpful and can distort learning.\nLoss curves spike or crash. Generated samples look worse over time instead of better, making training unstable.\nMode collapse and exploding gradients can occur.\nHow can we solve this problem?\nLet's introduce a 1-Lipschitz Function. This will solve our problem.\nWe will add a 1-Lipschitz Function to the critic (discriminator).\nIn the Lipshitz function, we will give any 2 images (x1 and x2) as input.\nNumerator: Absolute difference between the critic predictions\nDenominator: Average pixelwise absolute difference between two images\n.\nWhat we are trying to do with this function is to limit the rate at which the predictions of the critic can change between 2 images, when comparing it with the real/pixel-wise difference of the image.\nYou can get this using a graph by visualizing 2 (white) cones whose origin can be moved along the graph so that the whole graph always stays outside the cones.\nBy doing so, we are clipping the weights of the critic to lie within a small range, eliminating problems of exploding gradients.\nBut again NEW PROBLEM !!\nLearning significantly decreases 😢 😢\nA much better Solution is the WGAN-GP\nAdding Gradient Penalty\nAs the Lipshitz function did a partial job. We need to modify/tweak the function so that it can eliminate the problem of\nLearning significantly decreases\n.\nThe solution should be to penalize the Gradient if it's too far away from 1.\nThe Lipschitz constraint is enforced by including a gradient penalty (GP) term in the loss function for the critic. The GP penalizes the model if the gradient norm deviates from 1\nThe GP term, which we will be using with Lipschitz constraint, will measure the square difference between the gradient of the prediction wrt input image and 1.\nFrom the above image, it's clear that the gradient penalty increases as we move away from 1.\nThere are a few important things to take care of while using training WGAN-GP\n:\nWith Wasserstein loss, the Critic must be trained to converge before updating the Generator. To do so, we train the Critic several times between Generator updates. A typical ratio used is\n3 to 5 Critic updates per Generator update\n.\nBatch Normalization shouldn't be used in the critic. Batch Norm creates a correlation between images in the same batch. This makes the gradient penalty loss less effective. Experiments have shown that WGAN-GPs works great without it anyway\nResults after WGAN-GP model:\nUsing Wasserstein distance + Lipschitz function + Gradient penalty has improved the results of the images generated significantly.\nTraditional GANs suffer from vanishing gradients or mode collapse due to the Jensen-Shannon divergence. WGAN-GP, by using Wasserstein distance + gradient penalty, leads to more consistent and stable convergence. You get smoother generator updates, making training less sensitive to hyperparameters and architectural choices.\nIn classic GANs, generators can produce a limited variety (mode collapse). WGAN-GP encourages the generator to explore the data distribution more fully, thanks to the meaningful gradient feedback from the critic. Hence, more diverse outputs, especially noticeable in image generation tasks.\nThe loss correlates with sample quality, unlike traditional GAN loss, which becomes meaningless when the discriminator is too good. Hence, you can track training progress numerically, not just visually\nThere are no sudden spikes or flat zones in gradients. Hence, more efficient and productive generator learning.\nDisadvantages of WGAN\nSlower training, higher memory usage, and increased training time, especially on large datasets.\nHyperparameters are sensitive. Even a small change in hyperparameters can cause a lot of change. It needs to be tuned carefully.\nSummarize\nA WGAN-GP uses the Wasserstein loss\nThe WGAN-GP is trained using labels of 1 for real and -1 for fake\nNo Sigmoid activation in the final layer of the Critic\nInclude a GP term in the loss function for the Critic\nTrain the\nCritic multiple times\nfor each update of the Generator\nThere are\nno Batch Norm layers\nin the Critic", "metadata": { "title": "GEN-AI 5 : WGAN and WGAN-GP - DEV Community", "description": "Why ❌ GAN In the ever-evolving landscape of generative models, GANs have taken center... Tagged with generative, deeplearning, wgan, wgangp.", "keywords": "generative, deeplearning, wgan, wgangp, software, coding, development, engineering, inclusive, community", "author": null, "og:type": "article", "og:url": "https://dev.to/mitanshgor/gen-ai-2-generative-ai-and-auto-encoders-do8", "og:title": "GEN-AI 5 : WGAN and WGAN-GP", "og:description": "Why ❌ GAN In the ever-evolving landscape of generative models, GANs have taken center...", "og:site_name": "DEV Community", "og:image": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fd1i5qa2ukurz11tb0q41.jpeg", "twitter:site": "@thepracticaldev", "twitter:creator": "@", "twitter:title": "GEN-AI 5 : WGAN and WGAN-GP", "twitter:description": "Why ❌ GAN In the ever-evolving landscape of generative models, GANs have taken center...", "twitter:card": "summary_large_image", "twitter:widgets:new-embed-design": "on", "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1000,height=500,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fd1i5qa2ukurz11tb0q41.jpeg" }, "paywall": { "listed_paid": false, "detected": false, "marker": "", "public_only": false, "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙" }, "crawl": { "success": true, "status_code": 200, "error": "" }, "code_blocks": [], "images": [ { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyi492wjs6nr3z58wf2hg.png", "bytes": 290026, "reused": true, "local_path": "attachments/a627609d3fba/images/002.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyi492wjs6nr3z58wf2hg.png", "alt": "I0", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkddlg8za5taek2wm2503.gif", "bytes": 62844, "reused": true, "local_path": "attachments/a627609d3fba/images/003.gif", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkddlg8za5taek2wm2503.gif", "alt": "I1", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5lout24z5bpr2hnaw7vf.png", "bytes": 95468, "reused": true, "local_path": "attachments/a627609d3fba/images/004.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5lout24z5bpr2hnaw7vf.png", "alt": "I2", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fdia5wcuesljnfdvcm59n.png", "bytes": 35586, "reused": true, "local_path": "attachments/a627609d3fba/images/005.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fdia5wcuesljnfdvcm59n.png", "alt": "I3", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fu2hxwngfpuhab0mlsvf7.png", "bytes": 53088, "reused": true, "local_path": "attachments/a627609d3fba/images/006.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fu2hxwngfpuhab0mlsvf7.png", "alt": "I4", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpqyaa5r2vhscshwtdqeh.png", "bytes": 8718, "reused": true, "local_path": "attachments/a627609d3fba/images/007.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpqyaa5r2vhscshwtdqeh.png", "alt": "I5", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fip68lijcqaz11cb3ytl7.gif", "bytes": 435498, "reused": true, "local_path": "attachments/a627609d3fba/images/008.gif", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fip68lijcqaz11cb3ytl7.gif", "alt": "I6", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Futcfv5rhwinwjilaquer.png", "bytes": 84232, "reused": true, "local_path": "attachments/a627609d3fba/images/009.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Futcfv5rhwinwjilaquer.png", "alt": "I7", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fevcfujtd57fj59oku5c2.png", "bytes": 7196, "reused": true, "local_path": "attachments/a627609d3fba/images/010.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fevcfujtd57fj59oku5c2.png", "alt": "I8", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ft56qfswe9xzp17xwgvij.png", "bytes": 90052, "reused": true, "local_path": "attachments/a627609d3fba/images/011.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ft56qfswe9xzp17xwgvij.png", "alt": "III", "downloaded": true }, { "status": "ok", "url": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ffarqvyv498oeh1bkku1r.png", "bytes": 296668, "reused": true, "local_path": "attachments/a627609d3fba/images/012.png", "src": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ffarqvyv498oeh1bkku1r.png", "alt": "II", "downloaded": true } ], "audio": [], "videos": [], "links": [ { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fyi492wjs6nr3z58wf2hg.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fkddlg8za5taek2wm2503.gif", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5lout24z5bpr2hnaw7vf.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fdia5wcuesljnfdvcm59n.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fu2hxwngfpuhab0mlsvf7.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpqyaa5r2vhscshwtdqeh.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fip68lijcqaz11cb3ytl7.gif", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Futcfv5rhwinwjilaquer.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fevcfujtd57fj59oku5c2.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ft56qfswe9xzp17xwgvij.png", "label": "", "type": "a" }, { "href": "https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ffarqvyv498oeh1bkku1r.png", "label": "", "type": "a" } ], "repair_version": 2, "repaired_at": "2026-10-14T10:48:48Z", "source_url": "https://dev.to/mitanshgor/gen-ai-2-generative-ai-and-auto-encoders-do8", "papers": [], "actual_video_files": [], "body_chars": 7894, "expected_body_images": 11, "discovery_status": "ok", "delivery_status": "ok", "missing_reason": "", "raw_html": "attachments/a627609d3fba/pages/source.html", "local_html": "attachments/a627609d3fba/pages/article.html", "body_base_path": "attachments/a627609d3fba/pages/", "body_markdown_path": "attachments/a627609d3fba/pages/article.md" }