{
  "source_file": "0930",
  "source_platform": "个人博客",
  "author": "Christopher Olah / colah's blog",
  "title_listed": "Calculus on Computational Graphs: Backpropagation",
  "keywords": "反向传播,计算图,微积分",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "否",
  "url": "https://colah.github.io/posts/2015-08-Backprop/",
  "publish_date": "2015-08",
  "corpus_stage": "预训练/SFT",
  "notes": "反向传播与计算图经典教程",
  "id": "9271f78b6d8e",
  "collected_at": "2026-10-02T14:28:13",
  "title": "Calculus on Computational Graphs: Backpropagation -- colah's blog",
  "body_markdown": "Calculus on Computational Graphs: Backpropagation\nPosted on August 31, 2015\nIntroduction\nBackpropagation is the key algorithm that makes training deep models computationally tractable. For modern neural networks, it can make training with gradient descent as much as ten million times faster, relative to a naive implementation. That’s the difference between a model taking a week to train and taking 200,000 years.\nBeyond its use in deep learning, backpropagation is a powerful computational tool in many other areas, ranging from weather forecasting to analyzing numerical stability – it just goes by different names. In fact, the algorithm has been reinvented at least dozens of times in different fields (see\nGriewank (2010)\n). The general, application independent, name is “reverse-mode differentiation.”\nFundamentally, it’s a technique for calculating derivatives quickly. And it’s an essential trick to have in your bag, not only in deep learning, but in a wide variety of numerical computing situations.\nComputational Graphs\nComputational graphs are a nice way to think about mathematical expressions. For example, consider the expression\n\\(e=(a+b)*(b+1)\\)\n. There are three operations: two additions and one multiplication. To help us talk about this, let’s introduce two intermediary variables,\n\\(c\\)\nand\n\\(d\\)\nso that every function’s output has a variable. We now have:\n\\[c=a+b\\]\n\\[d=b+1\\]\n\\[e=c*d\\]\nTo create a computational graph, we make each of these operations, along with the input variables, into nodes. When one node’s value is the input to another node, an arrow goes from one to another.\nThese sorts of graphs come up all the time in computer science, especially in talking about functional programs. They are very closely related to the notions of dependency graphs and call graphs. They’re also the core abstraction behind the popular deep learning framework\nTheano\n.\nWe can evaluate the expression by setting the input variables to certain values and computing nodes up through the graph. For example, let’s set\n\\(a=2\\)\nand\n\\(b=1\\)\n:\nThe expression evaluates to\n\\(6\\)\n.\nDerivatives on Computational Graphs\nIf one wants to understand derivatives in a computational graph, the key is to understand derivatives on the edges. If\n\\(a\\)\ndirectly affects\n\\(c\\)\n, then we want to know how it affects\n\\(c\\)\n. If\n\\(a\\)\nchanges a little bit, how does\n\\(c\\)\nchange? We call this the\npartial derivative\nof\n\\(c\\)\nwith respect to\n\\(a\\)\n.\nTo evaluate the partial derivatives in this graph, we need the\nsum rule\nand the\nproduct rule\n:\n\\[\\frac{\\partial}{\\partial a}(a+b) = \\frac{\\partial a}{\\partial a} + \\frac{\\partial b}{\\partial a} = 1\\]\n\\[\\frac{\\partial}{\\partial u}uv = u\\frac{\\partial v}{\\partial u} + v\\frac{\\partial u}{\\partial u} = v\\]\nBelow, the graph has the derivative on each edge labeled.\nWhat if we want to understand how nodes that aren’t directly connected affect each other? Let’s consider how\n\\(e\\)\nis affected by\n\\(a\\)\n. If we change\n\\(a\\)\nat a speed of 1,\n\\(c\\)\nalso changes at a speed of\n\\(1\\)\n. In turn,\n\\(c\\)\nchanging at a speed of\n\\(1\\)\ncauses\n\\(e\\)\nto change at a speed of\n\\(2\\)\n. So\n\\(e\\)\nchanges at a rate of\n\\(1*2\\)\nwith respect to\n\\(a\\)\n.\nThe general rule is to sum over all possible paths from one node to the other, multiplying the derivatives on each edge of the path together. For example, to get the derivative of\n\\(e\\)\nwith respect to\n\\(b\\)\nwe get:\n\\[\\frac{\\partial e}{\\partial b}= 1*2 + 1*3\\]\nThis accounts for how b affects e through c and also how it affects it through d.\nThis general “sum over paths” rule is just a different way of thinking about the\nmultivariate chain rule\n.\nFactoring Paths\nThe problem with just “summing over the paths” is that it’s very easy to get a combinatorial explosion in the number of possible paths.\nIn the above diagram, there are three paths from\n\\(X\\)\nto\n\\(Y\\)\n, and a further three paths from\n\\(Y\\)\nto\n\\(Z\\)\n. If we want to get the derivative\n\\(\\frac{\\partial Z}{\\partial X}\\)\nby summing over all paths, we need to sum over\n\\(3*3 = 9\\)\npaths:\n\\[\\frac{\\partial Z}{\\partial X} = \\alpha\\delta + \\alpha\\epsilon + \\alpha\\zeta + \\beta\\delta + \\beta\\epsilon + \\beta\\zeta + \\gamma\\delta + \\gamma\\epsilon + \\gamma\\zeta\\]\nThe above only has nine paths, but it would be easy to have the number of paths to grow exponentially as the graph becomes more complicated.\nInstead of just naively summing over the paths, it would be much better to factor them:\n\\[\\frac{\\partial Z}{\\partial X} = (\\alpha + \\beta + \\gamma)(\\delta + \\epsilon + \\zeta)\\]\nThis is where “forward-mode differentiation” and “reverse-mode differentiation” come in. They’re algorithms for efficiently computing the sum by factoring the paths. Instead of summing over all of the paths explicitly, they compute the same sum more efficiently by merging paths back together at every node. In fact, both algorithms touch each edge exactly once!\nForward-mode differentiation starts at an input to the graph and moves towards the end. At every node, it sums all the paths feeding in. Each of those paths represents one way in which the input affects that node. By adding them up, we get the total way in which the node is affected by the input, it’s derivative.\nThough you probably didn’t think of it in terms of graphs, forward-mode differentiation is very similar to what you implicitly learned to do if you took an introduction to calculus class.\nReverse-mode differentiation, on the other hand, starts at an output of the graph and moves towards the beginning. At each node, it merges all paths which originated at that node.\nForward-mode differentiation tracks how one input affects every node. Reverse-mode differentiation tracks how every node affects one output. That is, forward-mode differentiation applies the operator\n\\(\\frac{\\partial}{\\partial X}\\)\nto every node, while reverse mode differentiation applies the operator\n\\(\\frac{\\partial Z}{\\partial}\\)\nto every node.\n1\nComputational Victories\nAt this point, you might wonder why anyone would care about reverse-mode differentiation. It looks like a strange way of doing the same thing as the forward-mode. Is there some advantage?\nLet’s consider our original example again:\nWe can use forward-mode differentiation from\n\\(b\\)\nup. This gives us the derivative of every node with respect to\n\\(b\\)\n.\nWe’ve computed\n\\(\\frac{\\partial e}{\\partial b}\\)\n, the derivative of our output with respect to one of our inputs.\nWhat if we do reverse-mode differentiation from\n\\(e\\)\ndown? This gives us the derivative of\n\\(e\\)\nwith respect to every node:\nWhen I say that reverse-mode differentiation gives us the derivative of e with respect to every node, I really do mean\nevery node\n. We get both\n\\(\\frac{\\partial e}{\\partial a}\\)\nand\n\\(\\frac{\\partial e}{\\partial b}\\)\n, the derivatives of\n\\(e\\)\nwith respect to both inputs. Forward-mode differentiation gave us the derivative of our output with respect to a single input, but reverse-mode differentiation gives us all of them.\nFor this graph, that’s only a factor of two speed up, but imagine a function with a million inputs and one output. Forward-mode differentiation would require us to go through the graph a million times to get the derivatives. Reverse-mode differentiation can get them all in one fell swoop! A speed up of a factor of a million is pretty nice!\nWhen training neural networks, we think of the cost (a value describing how bad a neural network performs) as a function of the parameters (numbers describing how the network behaves). We want to calculate the derivatives of the cost with respect to all the parameters, for use in\ngradient descent\n. Now, there’s often millions, or even tens of millions of parameters in a neural network. So, reverse-mode differentiation, called backpropagation in the context of neural networks, gives us a massive speed up!\n(Are there any cases where forward-mode differentiation makes more sense? Yes, there are! Where the reverse-mode gives the derivatives of one output with respect to all inputs, the forward-mode gives us the derivatives of all outputs with respect to one input. If one has a function with lots of outputs, forward-mode differentiation can be much, much, much faster.)\nIsn’t This Trivial?\nWhen I first understood what backpropagation was, my reaction was: “Oh, that’s just the chain rule! How did it take us so long to figure out?” I’m not the only one who’s had that reaction. It’s true that if you ask “is there a smart way to calculate derivatives in feedforward neural networks?” the answer isn’t that difficult.\nBut I think it was much more difficult than it might seem. You see, at the time backpropagation was invented, people weren’t very focused on the feedforward neural networks that we study. It also wasn’t obvious that derivatives were the right way to train them. Those are only obvious once you realize you can quickly calculate derivatives. There was a circular dependency.\nWorse, it would be very easy to write off any piece of the circular dependency as impossible on casual thought. Training neural networks with derivatives? Surely you’d just get stuck in local minima. And obviously it would be expensive to compute all those derivatives. It’s only because we know this approach works that we don’t immediately start listing reasons it’s likely not to.\nThat’s the benefit of hindsight. Once you’ve framed the question, the hardest work is already done.\nConclusion\nDerivatives are cheaper than you think. That’s the main lesson to take away from this post. In fact, they’re unintuitively cheap, and us silly humans have had to repeatedly rediscover this fact. That’s an important thing to understand in deep learning. It’s also a really useful thing to know in other fields, and only more so if it isn’t common knowledge.\nAre there other lessons? I think there are.\nBackpropagation is also a useful lens for understanding how derivatives flow through a model. This can be extremely helpful in reasoning about why some models are difficult to optimize. The classic example of this is the problem of vanishing gradients in recurrent neural networks.\nFinally, I claim there is a broad algorithmic lesson to take away from these techniques. Backpropagation and forward-mode differentiation use a powerful pair of tricks (linearization and dynamic programming) to compute derivatives more efficiently than one might think possible. If you really understand these techniques, you can use them to efficiently calculate several other interesting expressions involving derivatives. We’ll explore this in a later blog post.\nThis post gives a very abstract treatment of backpropagation. I strongly recommend reading Michael Nielsen’s\nchapter on it\nfor an excellent discussion, more concretely focused on neural networks.\nAcknowledgments\nThank you to\nGreg Corrado\n,\nJon Shlens\n,\nSamy Bengio\nand\nAnelia Angelova\nfor taking the time to proofread this post.\nThanks also to\nDario Amodei\n,\nMichael Nielsen\nand\nYoshua Bengio\nfor discussion of approaches to explaining backpropagation. Also thanks to all those who tolerated me practicing explaining backpropagation in talks and seminar series!\nThis might feel a bit like\ndynamic programming\n. That’s because it is!\n↩\nMore Posts\nUnderstanding Convolutions\nUnderstanding LSTM Networks\nVisualizing MNIST\nAn Exploration of Dimensionality Reduction\nConv Nets\nA Modular Perspective\nBuilt by\nOinkina\nwith\nHakyll\nusing\nBootstrap\n,\nMathJax\n,\nDisqus\n,\nMathBox.js\n,\nHighlight.js\n, \n                    and\nFootnotes.js\n.\nEnable JavaScript for footnotes, Disqus comments, and other cool stuff.",
  "metadata": {
    "title": "Calculus on Computational Graphs: Backpropagation -- colah's blog",
    "description": null,
    "keywords": null,
    "author": null
  },
  "paywall": {
    "listed_paid": false,
    "detected": false,
    "marker": "",
    "public_only": false,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [],
  "images": [
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Backprop/img/tree-def.png",
      "bytes": 78900,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-f7e83edb359e8346168a.png",
      "src": "https://colah.github.io/posts/2015-08-Backprop/img/tree-def.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Backprop/img/tree-eval.png",
      "bytes": 91995,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-0767b38d3b0979177888.png",
      "src": "https://colah.github.io/posts/2015-08-Backprop/img/tree-eval.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Backprop/img/tree-eval-derivs.png",
      "bytes": 98591,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-53eda9f3efc74a54efc1.png",
      "src": "https://colah.github.io/posts/2015-08-Backprop/img/tree-eval-derivs.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Backprop/img/chain-def-greek.png",
      "bytes": 66505,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-15dd7334c656288f035c.png",
      "src": "https://colah.github.io/posts/2015-08-Backprop/img/chain-def-greek.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Backprop/img/chain-forward-greek.png",
      "bytes": 107493,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-aef83e0a8715d581c034.png",
      "src": "https://colah.github.io/posts/2015-08-Backprop/img/chain-forward-greek.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Backprop/img/chain-backward-greek.png",
      "bytes": 107708,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-6707042d77ec53c35248.png",
      "src": "https://colah.github.io/posts/2015-08-Backprop/img/chain-backward-greek.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Backprop/img/tree-forwradmode.png",
      "bytes": 108036,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-712124d0be87f41672e2.png",
      "src": "https://colah.github.io/posts/2015-08-Backprop/img/tree-forwradmode.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2015-08-Backprop/img/tree-backprop.png",
      "bytes": 108293,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-8f85b3ede2b16bbfd8b0.png",
      "src": "https://colah.github.io/posts/2015-08-Backprop/img/tree-backprop.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2014-07-Understanding-Convolutions/img/fig.png",
      "bytes": 111007,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-220e2c47837c494432a0.png",
      "src": "https://colah.github.io/posts/2014-07-Understanding-Convolutions/img/fig.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/images/post-covers/lstm.png",
      "bytes": 43204,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-eb62dd87e162d841449c.png",
      "src": "https://colah.github.io/images/post-covers/lstm.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2014-10-Visualizing-MNIST/img/fig.png",
      "bytes": 145523,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-a5fbfc47f6253032212f.png",
      "src": "https://colah.github.io/posts/2014-10-Visualizing-MNIST/img/fig.png",
      "alt": "",
      "downloaded": true
    },
    {
      "status": "ok",
      "url": "https://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/fig.png",
      "bytes": 38963,
      "reused": false,
      "local_path": "attachments/9271f78b6d8e/images/body-4e9777debd9700c02b1f.png",
      "src": "https://colah.github.io/posts/2014-07-Conv-Nets-Modular/img/fig.png",
      "alt": "",
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [],
  "links": [
    {
      "href": "http://www.math.uiuc.edu/documenta/vol-ismp/52_griewank-andreas-b.pdf",
      "label": "Griewank (2010)",
      "type": "a"
    },
    {
      "href": "http://deeplearning.net/software/theano/",
      "label": "Theano",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Partial_derivative",
      "label": "partial derivative",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Sum_rule_in_differentiation",
      "label": "sum rule",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Product_rule",
      "label": "product rule",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Chain_rule",
      "label": "multivariate chain rule",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Gradient_descent",
      "label": "gradient descent",
      "type": "a"
    },
    {
      "href": "http://neuralnetworksanddeeplearning.com/chap2.html",
      "label": "chapter on it",
      "type": "a"
    },
    {
      "href": "http://research.google.com/pubs/GregCorrado.html",
      "label": "Greg Corrado",
      "type": "a"
    },
    {
      "href": "https://shlens.wordpress.com/",
      "label": "Jon Shlens",
      "type": "a"
    },
    {
      "href": "http://bengio.abracadoudou.com/",
      "label": "Samy Bengio",
      "type": "a"
    },
    {
      "href": "http://www.vision.caltech.edu/anelia/",
      "label": "Anelia Angelova",
      "type": "a"
    },
    {
      "href": "https://www.linkedin.com/pub/dario-amodei/4/493/393",
      "label": "Dario Amodei",
      "type": "a"
    },
    {
      "href": "http://michaelnielsen.org/",
      "label": "Michael Nielsen",
      "type": "a"
    },
    {
      "href": "http://www.iro.umontreal.ca/~bengioy/yoshua_en/index.html",
      "label": "Yoshua Bengio",
      "type": "a"
    },
    {
      "href": "https://en.wikipedia.org/wiki/Dynamic_programming",
      "label": "dynamic programming",
      "type": "a"
    },
    {
      "href": "https://colah.github.io/posts/2014-07-Understanding-Convolutions/",
      "label": "Understanding Convolutions",
      "type": "a"
    },
    {
      "href": "https://colah.github.io/posts/2015-08-Understanding-LSTMs/",
      "label": "Understanding LSTM Networks",
      "type": "a"
    },
    {
      "href": "https://colah.github.io/posts/2014-10-Visualizing-MNIST/",
      "label": "Visualizing MNIST An Exploration of Dimensionality Reduction",
      "type": "a"
    },
    {
      "href": "https://colah.github.io/posts/2014-07-Conv-Nets-Modular/",
      "label": "Conv Nets A Modular Perspective",
      "type": "a"
    },
    {
      "href": "https://github.com/oinkina",
      "label": "Oinkina",
      "type": "a"
    },
    {
      "href": "http://jaspervdj.be/hakyll",
      "label": "Hakyll",
      "type": "a"
    },
    {
      "href": "http://getbootstrap.com/",
      "label": "Bootstrap",
      "type": "a"
    },
    {
      "href": "http://www.mathjax.org/",
      "label": "MathJax",
      "type": "a"
    },
    {
      "href": "http://disqus.com/",
      "label": "Disqus",
      "type": "a"
    },
    {
      "href": "https://github.com/unconed/MathBox.js",
      "label": "MathBox.js",
      "type": "a"
    },
    {
      "href": "http://highlightjs.org/",
      "label": "Highlight.js",
      "type": "a"
    },
    {
      "href": "http://ignorethecode.net/blog/2010/04/20/footnotes/",
      "label": "Footnotes.js",
      "type": "a"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T10:52:33Z",
  "source_url": "https://colah.github.io/posts/2015-08-Backprop/",
  "papers": [
    {
      "status": "download_error",
      "url": "http://www.math.uiuc.edu/documenta/vol-ismp/52_griewank-andreas-b.pdf",
      "reason": "ConnectionError: HTTPSConnectionPool(host='math.illinois.edudocumenta', port=443): Max retries exceeded with url: /vol-ismp/52_griewank-andreas-b.pdf (Caused by NameResolutionError(\"HTTPSConnection",
      "retried_at": "2026-10-20T22:24:48Z"
    }
  ],
  "actual_video_files": [],
  "body_chars": 11537,
  "expected_body_images": 12,
  "discovery_status": "ok",
  "delivery_status": "partial",
  "missing_reason": "download_error",
  "raw_html": "attachments/9271f78b6d8e/pages/source.html",
  "local_html": "attachments/9271f78b6d8e/pages/article.html",
  "body_base_path": "attachments/9271f78b6d8e/pages/",
  "body_markdown_path": "attachments/9271f78b6d8e/pages/article.md"
}