{
  "source_file": "0930",
  "source_platform": "Dev.to",
  "author": "Priyesh Dave",
  "title_listed": "Day 19: The Vanishing Gradient Problem",
  "keywords": "",
  "domain": "机器学习",
  "content_type": "技术教程",
  "is_paid_flag": "否",
  "url": "https://dev.to/priyeshdave6/day-19-the-vanishing-gradient-problem-128i",
  "publish_date": "2026-09-15",
  "corpus_stage": "",
  "notes": "",
  "id": "4883efe6350f",
  "collected_at": "2026-10-02T13:49:34",
  "title": "Day 19: The Vanishing Gradient Problem - DEV Community",
  "body_markdown": "What Is the Vanishing Gradient Problem?\nThe vanishing gradient problem makes it difficult to train very deep neural networks. It happens when the “gradient”—the math signal that tells each layer how to improve—gets smaller and smaller as it moves backward through the network. This especially affects models for sequence data, like Recurrent Neural Networks (RNNs).\nA\ngradient\nis a number measuring how much changing a model parameter (like a weight) will change the model’s error. Large gradients mean weights should change a lot. Small gradients mean they barely change at all.\nWhen these gradients become very small, early layers in the network stop learning. The whole model struggles to improve.\nWhy Gradients Matter in Deep Learning\nGradients are how neural networks learn. Training works by making predictions, measuring how wrong they are, and adjusting the weights to make things less wrong next time.\nPractically, you run some data through the network, compute an error, and use calculus (via derivatives) to find the gradient.\nBackpropagation\nis the algorithm that efficiently computes these gradients backward through the network, from the output layer back to the first layer.\nYou can think of each weight in the network as a dial you can turn. The gradient tells you which way, and how much, to turn that dial to reduce error. If the gradient is almost zero, the dial barely moves—no learning happens.\nHow Vanishing Gradients Happen\nDeep networks chain together multiple layers. Each layer passes its outputs through an\nactivation function\n. An activation function is just a small math operation (like sigmoid or tanh) that helps the network model complex relationships. Functions like sigmoid and tanh “squash” their inputs into a narrow range—sigmoid to (0, 1); tanh to (-1, 1).\nDuring backpropagation, gradients flow backward through these activation functions. At each step, the gradient gets multiplied by the slope (derivative) of the activation function at that point. For sigmoid or tanh, this multiplying factor is always less than 1.\nImagine every layer in a network multiplies the gradient by 0.4 (a typical value for tanh or sigmoid). If you have 5 layers:\nAfter 1 layer: original × 0.4\nAfter 2 layers: original × 0.16\nAfter 5 layers: original × 0.4⁵ ≈ original × 0.01\nBy the time the gradient reaches the earliest layers, it's nearly zero.\nAnalogy:\nImagine whispering a message down a long line of people. If each person hears only half as loud as the last, the message fades to almost nothing before reaching the end. This is how gradients can fade in deep networks.\nConsequences in Sequence Models\nSequence models like RNNs process data that stretches out in time—like words in a sentence or steps in a time series. RNNs use the same set of weights, over and over, for each time step.\nWith vanishing gradients, learning signals must travel backward through many time steps. The longer the sequence and the more steps, the greater the problem.\nIf your RNN needs to predict a word at the sentence end based on a word from the beginning (maybe 20 steps ago), the gradient must travel back through all those steps. If it shrinks at every step, it will be nearly zero by the time it reaches the start. As a result, the network almost can’t learn to connect early and late information. Early RNNs struggled to model long-range relationships for this reason.\nVisualizing the Effect\nHere’s a concrete math example. Suppose you have 6 layers, each using tanh. Tanh’s maximum slope (derivative) is about 1, but typical values are smaller—roughly 0.4 to 0.5 for most inputs.\nIf, at each layer, the gradient is multiplied by 0.5:\nStart: 1.0\nAfter 1 layer: 0.5\nAfter 2 layers: 0.25\nAfter 3 layers: 0.125\nAfter 4 layers: 0.0625\nAfter 5 layers: 0.03125\nAfter 6 layers: 0.015625\nAt layer 6, less than 2% of the original gradient remains. The optimizer can barely adjust these early weights.\nPyTorch Example: Gradients Shrinking in Deep Tanh Networks\nBelow is a runnable code example. This builds a deep feedforward neural network with tanh activations. After doing one forward and backward pass on random data, it prints the average absolute gradient for each layer. You’ll see that earlier layers (closer to the input) have much smaller gradients than later layers (closer to the output).\nimport\ntorch\nimport\ntorch.nn\nas\nnn\n# Deep network: 7 hidden layers, all tanh\nclass\nDeepTanhNet\n(\nnn\n.\nModule\n):\ndef\n__init__\n(\nself\n,\ndim\n,\ndepth\n):\nsuper\n().\n__init__\n()\nlayers\n=\n[]\nfor\n_\nin\nrange\n(\ndepth\n):\nlayers\n.\nappend\n(\nnn\n.\nLinear\n(\ndim\n,\ndim\n))\nlayers\n.\nappend\n(\nnn\n.\nTanh\n())\nself\n.\nseq\n=\nnn\n.\nSequential\n(\n*\nlayers\n)\nself\n.\nout\n=\nnn\n.\nLinear\n(\ndim\n,\n1\n)\ndef\nforward\n(\nself\n,\nx\n):\nreturn\nself\n.\nout\n(\nself\n.\nseq\n(\nx\n))\ndepth\n=\n7\ndim\n=\n8\nnet\n=\nDeepTanhNet\n(\ndim\n,\ndepth\n)\nx\n=\ntorch\n.\nrandn\n(\n1\n,\ndim\n)\ntarget\n=\ntorch\n.\ntensor\n([[\n0.0\n]])\nout\n=\nnet\n(\nx\n)\nloss\n=\n(\nout\n-\ntarget\n).\npow\n(\n2\n).\nmean\n()\nloss\n.\nbackward\n()\n# Print average absolute gradient for each Linear layer\nfor\ni\n,\nlayer\nin\nenumerate\n(\nnet\n.\nseq\n):\nif\nisinstance\n(\nlayer\n,\nnn\n.\nLinear\n):\ngrad\n=\nlayer\n.\nweight\n.\ngrad\n.\nabs\n().\nmean\n().\nitem\n()\nprint\n(\nf\n\"\nLayer\n{\ni\n//\n2\n+\n1\n}\nmean abs gradient:\n{\ngrad\n:\n.\n6\nf\n}\n\"\n)\nEnter fullscreen mode\nExit fullscreen mode\nThe printed gradients will be much smaller for early layers than for those near the output. This is the vanishing gradient in action.\nCommon Ways to Address the Vanishing Gradient Problem\nSeveral well-established tools help with vanishing gradients:\nReLU Activations:\nThe Rectified Linear Unit (ReLU) activation doesn’t squash values for positive inputs, so gradients are less likely to vanish.\nLSTM and GRU Units:\nIn RNNs, specially designed units like Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) let gradients flow more easily across many time steps.\nBetter Weight Initialization:\nInitializing network weights in a smart way can keep gradients in a manageable range.\nBatch Normalization:\nBatch normalization helps keep values stable as they flow through the network, indirectly preventing gradients from shrinking too much.\nEach of these approaches has trade-offs. All will be covered in more detail later in this series. For now, remember: vanishing gradients are a key challenge for deep and sequential neural networks, but there are proven solutions.\nKey Takeaways\nGradients indicate how much model weights should change to reduce error.\nVanishing gradients occur when gradients become extremely small in early layers of deep networks.\nActivation functions like sigmoid and tanh contribute to gradient shrinking through repeated multiplication by small derivatives.\nThis issue makes it hard for models, especially RNNs, to learn long-range dependencies.\nTechniques such as ReLU activations, LSTM/GRU units, batch normalization, and better weight initialization help mitigate vanishing gradients.\nTry It Yourself\nBuild a simple multi-layer perceptron (MLP) with several hidden layers, using tanh activations, in your favorite deep learning framework. Pass some random data through the network and perform a backward pass. Print out the average absolute gradients for each layer and observe how the gradients are much smaller in earlier layers compared to later ones.\nFurther Resources\n🎥\nStanford CS224N Lecture 7 – Vanishing Gradients, Fancy RNNs\n📄\nVizLearn – Vanishing Gradient Problem in RNN\n🎥\nMIT Introduction to Deep Learning – Sequence Modeling with Neural Networks\n📄\nMichael Brenndoerfer – Vanishing Gradients: Why RNNs Fail on Long Sequences\n📄\nHu et al., Overcoming the vanishing gradient problem in plain recurrent networks (arXiv)\nComing up on Day 20:\nLSTM Networks Explained",
  "metadata": {
    "title": "Day 19: The Vanishing Gradient Problem - DEV Community",
    "description": "What Is the Vanishing Gradient Problem?   The vanishing gradient problem makes it difficult... Tagged with ai, nlp, genai, 100daysofcode.",
    "keywords": "ai, nlp, genai, 100daysofcode, software, coding, development, engineering, inclusive, community",
    "author": null,
    "og:type": "article",
    "og:url": "https://dev.to/priyeshdave6/day-19-the-vanishing-gradient-problem-128i",
    "og:title": "Day 19: The Vanishing Gradient Problem",
    "og:description": "What Is the Vanishing Gradient Problem?   The vanishing gradient problem makes it difficult...",
    "og:site_name": "DEV Community",
    "og:image": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fz7eqzve1edsy1s650ki9.png",
    "twitter:site": "@thepracticaldev",
    "twitter:creator": "@",
    "twitter:title": "Day 19: The Vanishing Gradient Problem",
    "twitter:description": "What Is the Vanishing Gradient Problem?   The vanishing gradient problem makes it difficult...",
    "twitter:card": "summary_large_image",
    "twitter:widgets:new-embed-design": "on",
    "twitter:image:src": "https://media2.dev.to/dynamic/image/width=1200,height=627,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fz7eqzve1edsy1s650ki9.png"
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "file": "attachments/4883efe6350f/code/body-000.txt",
      "chars": 867,
      "language": "text"
    }
  ],
  "images": [],
  "audio": [],
  "videos": [
    {
      "url": "https://www.youtube.com/watch?v=QEw0qEa0E50",
      "src": "https://www.youtube.com/watch?v=QEw0qEa0E50",
      "status": "source_unavailable",
      "kind": "single_video",
      "error": "ERROR: [youtube] QEw0qEa0E50: Video unavailable\n",
      "title": "",
      "retried_at": "2026-10-20T18:01:50Z"
    }
  ],
  "links": [
    {
      "href": "https://www.youtube.com/watch?v=QEw0qEa0E50",
      "label": "Stanford CS224N Lecture 7 – Vanishing Gradients, Fancy RNNs",
      "type": "a"
    },
    {
      "href": "https://vizlearn.in/natural_language_processing/vanishing_gradient_problem_in_rnn.html",
      "label": "VizLearn – Vanishing Gradient Problem in RNN",
      "type": "a"
    },
    {
      "href": "https://www.classcentral.com/course/youtube-mit-6-s191-2018-sequence-modeling-with-neural-networks-128115",
      "label": "MIT Introduction to Deep Learning – Sequence Modeling with Neural Networks",
      "type": "a"
    },
    {
      "href": "https://mbrenndoerfer.com/writing/vanishing-gradients-rnn-long-range-dependencies",
      "label": "Michael Brenndoerfer – Vanishing Gradients: Why RNNs Fail on Long Sequences",
      "type": "a"
    },
    {
      "href": "https://arxiv.org/abs/1801.06105",
      "label": "Hu et al., Overcoming the vanishing gradient problem in plain recurrent networks (arXiv)",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:48:07Z",
  "source_url": "https://dev.to/priyeshdave6/day-19-the-vanishing-gradient-problem-128i",
  "papers": [
    {
      "status": "ok",
      "url": "https://arxiv.org/pdf/1801.06105",
      "bytes": 2276067,
      "reused": false,
      "local_path": "papers/repair/6a6488a3a73bd6e34747.pdf"
    }
  ],
  "actual_video_files": [],
  "body_chars": 7643,
  "expected_body_images": 0,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "source_unavailable",
  "raw_html": "attachments/4883efe6350f/pages/source.html",
  "local_html": "attachments/4883efe6350f/pages/article.html",
  "body_base_path": "attachments/4883efe6350f/pages/",
  "body_markdown_path": "attachments/4883efe6350f/pages/article.md"
}