{
  "source_file": "0930",
  "source_platform": "Medium",
  "author": "Ebrahim Pichka",
  "title_listed": "Automatic Differentiation (AutoDiff): A Brief Intro with Examples",
  "keywords": "自动微分,AutoDiff,深度学习",
  "domain": "机器学习",
  "content_type": "深度技术解析",
  "is_paid_flag": "部分免费/会员墙",
  "url": "https://towardsdatascience.com/automatic-differentiation-autodiff-a-brief-intro-with-examples-3f3d257ffe3b/",
  "publish_date": "2024-10",
  "corpus_stage": "预训练/SFT",
  "notes": "",
  "id": "dceb332dcd69",
  "collected_at": "2026-10-01T23:11:03",
  "title": "Automatic Differentiation (AutoDiff): A Brief Intro with Examples | Towards Data Science",
  "body_markdown": "Toggle Search[](https://towardsdatascience.com/)Toggle Mobile Navigation\n[](https://towardsdatascience.com/)\nPublish AI, ML & data-science insights to a global community of data professionals.\nSign in[Submit an Article](https://write.insightmediagroup.io)\n[Latest](https://towardsdatascience.com/latest/)[Editor's Picks](https://towardsdatascience.com/tag/editors-pick/)[Newsletter](https://towardsdatascience.com/tag/the-variable/)[ShipAI](https://towardsdatascience.com/shipai/)\n* * *\n[Write For TDS](https://towardsdatascience.com/submissions/)\n[LinkedIn](https://www.linkedin.com/company/towards-data-science/)[X](https://x.com/TDataScience)Toggle Search\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\n# Automatic Differentiation (AutoDiff): A Brief Intro with Examples\nAn introduction to the mechanics of AutoDiff, exploring its mathematical principles, implementation strategies, and applications\n[Ebrahim Pichka](https://towardsdatascience.com/author/ebrahimpichka/)\nOctober 11, 202411 min read\nin currently most-used frameworks\n![Photo by Bozhin Karaivanov on Unsplash](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/02zoQV7HydfU2dV8c-scaled.jpg)\nPhoto by [Bozhin Karaivanov](https://unsplash.com/@bkaraivanov?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com?utm_source=medium&utm_medium=referral)\nLearn this step by step with the interactive [Machine Learning](https://roadmap.sh/machine-learning) roadmap.\n## The Fundamental Role of Differentiation in Modern Machine Learning Optimization\nAt the heart of machine learning lies the optimization of loss/objective functions. This optimization process heavily relies on computing gradients of these functions with respect to model parameters. As Baydin et al. (2018) elucidate in their comprehensive survey [1], these gradients guide the iterative updates in optimization algorithms such as stochastic gradient descent (SGD):\n_θₜ₊₁ = θₜ - α ∇θ L(θₜ)_\nWhere:\n  * θₜ represents the model parameters at step t\n  * α is the learning rate\n  * ∇_θ L(θₜ) denotes the gradient of the loss function L with respect to the parameters θ\n\n\nThis simple update rule belies the complexity of computing gradients in deep neural networks with millions or even billions of parameters.\n* * *\n## 2. The Differentiation Triad\nDifferentiation can generally be performed in three main manners. **Symbolic** , **Numeric** , and **Automatic** Differentiation. We will now briefly discuss the differences between them.\n### 2.1 Symbolic Differentiation\n_Symbolic_ differentiation involves the manipulation of mathematical expressions to produce exact derivatives. If you have ever taken any introductory courses in calculus, this method must've been your first exposure to differentiation. While it provides precise results, it often leads to expression swell, making it impractical for the complex, nested functions typical in machine learning [1].\nConsider the function _f(x) = x⁴ + 3x² + 2x_. Symbolic differentiation would yield:\n_f'(x) = 4x³ + 6x + 2_\nWhile this is manageable for simple functions with clear analytical clsoed forms, imagine the complexity for a neural network with thousands of nonlinearities and multiple skip connections, branches, heads!\n### 2.2 Numeric Differentiation\nNumeric differentiation approximates derivatives using finite differences following thw formal definition of derivatives, namely:\n_f'(x) ≈ f(x + h) - f(x) / h_\nThis method simply tries to compute an approximate value for _f'_ by **assigning a** **very small value to** _**h**_ and computing the change it causes in the output of _f_. While straightforward to implement, it's realy susceptible to truncation errors (for large h) and round-off errors (for small h) [2]. Moreover, its computational cost scales poorly with the number of input variables as each input xᵢ would require calling of the function separately.\n### 2.3 Automatic Differentiation\nIn contrast with the two previous methods, Automatic Differentiation, _**AutoDiff**_ for short, _****_ strikes a balance between symbolic and numeric methods, computing exact derivatives (up to machine precision) efficiently by systematically applying **the chain rul** e to elementary operations and functions [1]. In short, [the chain rule](https://tutorial.math.lamar.edu/classes/calci/chainrule.aspx) basically says that th**e derivative of a composite functio** n is the **product of the derivatives of its componen** t functions. Mathematically, if we have two functions _y = f(u_) and _u = f(x_), we have:\n_**dy/dx = dy/du × du/dx**_\nwhere:\n  * _dy/dx_ is the derivative of _y_ with respect to _x_ (the overall derivative we're trying to find - in case of deep learning models, _y_ is usually the finall loss and _x_ is the doels weights)\n  * _dy/du_ is the derivative of _y_ with respect to an intermediate variable _u_\n  * _du/dx_ is the derivative of the intermediate variable _u_ with respect to _x_\n\n\nLeveraging the chain rule, along with some implementation details that we are going to discuss next, allows us to compute gradients in a very optimal manner.\n* * *\n## 3. AutoDiff Modes: Forward and Reverse\nAutoDiff can be practically done in two ways, namely **Forward** mode and **Reverse** mode differentiation, each having some computational advantages and disadvantages based on the use case.\n### 3.1 Forward Mode\nForward mode - also known as left-to-right - AutoDiff computes **directional** derivatives alongside the function evaluation. It's particularly efficient for functions with few inputs and many outputs [3].\nMathematically, for a function y = f(x) where x ∈ ℝⁿ and y ∈ ℝᵐ, forward mode computes the Jacobian-vector product on the side:\n![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/1wZ21Lgor0RqG1dHOOrSVHQ.png)\nW[here](https://maximerobeyns.com/of_vjps_and_jvps) _**J**_ is the Jacobian matrix and _**dot{x}**_ is the seed vector. For a detailed explanation of Jacobian-vector product see here.\nLet's implement a simple forward mode AD:\npython\n\n```\nclass Dual:    def __init__(self, value, derivative):        self.value = value        self.derivative = derivative # works as dot{x}    def __add__(self, other):        return Dual(self.value + other.value, self.derivative + other.derivative)    def __mul__(self, other):        return Dual(self.value * other.value,                     self.value * other.derivative + self.derivative * other.value)    def __pow__(self, n):        return Dual(self.value ** n,                     n * (self.value ** (n-1)) * self.derivative)def f(x):    return x**4 + 3*x**2 + 2*xx = Dual(2.0, 1.0)  # x = 2, dx/dx = 1 (~> dot{x})result = f(x)print(f\"f(2) = {result.value}, f'(2) = {result.derivative}\")# Output: f(2) = 42.0, f'(2) = 58.0\n```\n\nAs demonstrated, Forward autodiff augments each intermediate variable during evaluation of a function **with its derivative**. It involves replacing individual floating point values flowing through a function with **tuples of the original intermediate values also called primals paired with their derivatives**.\nTo compute the partial derivative of a function with respect to an input variable, we have to run **a separate forward pass** for each input variable of interest with corresponding seed set to 1. The forward mode autodiff produces one column of the corresponding Jacobian _**J.**_\nFor a two dimensional input _**x ∈ R²**_ , setting _**dot{x}**_ to [1, 0] yields the first column of _**J**_ which is the partial derivative w.r.t _x₁_ and setting it [0, 1] results in the second column which is the partial derivative w.r.t _x₂_.\nAri Seff does a great job explaining it in his [**AutoDiff video here**](https://youtu.be/wG_nF1awSSY?t=305).\n### 3.2 Reverse Mode\nReverse mode AutoDiff, which is the main AD method used in current major deep learning frameworks, computes gradients by propagating derivatives by **going backward** through the [**computation graph (see [6])**](https://simple-english-machine-learning.readthedocs.io/en/latest/neural-networks/computational-graphs.html)**starting from the output and then applying the chain rule until it traverses the whole graph**. It's particularly efficient for functions with many inputs and few outputs, which is the typical case in neural networks [3].\nReverse mode computes the vector-Jacobian product which is explained PyTorch's introduction to AtuoDiff in the \"**Vector Calculus using**`**autograd**`**\"** section [here](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html).\nHere's a simplified implementation of reverse mode AD:\npython\n\n```\nclass Node:    def __init__(self, value):        self.value = value        self.grad = 0        self._backward = lambda: None # this is defined as the forward mode is done based on the computation graph.         self._prev = set()    def __add__(self, other):        other = other if isinstance(other, Node) else Node(other)        out = Node(self.value + other.value)        out._prev = {self, other}        def _backward():            self.grad += out.grad            other.grad += out.grad        out._backward = _backward        return out    def __mul__(self, other):        other = other if isinstance(other, Node) else Node(other)        out = Node(self.value * other.value)        out._prev = {self, other}        def _backward():            self.grad += other.value * out.grad            other.grad += self.value * out.grad        out._backward = _backward        return out    def __pow__(self, n):        out = Node(self.value ** n)        out._prev = {self}        def _backward():            self.grad += n * (self.value ** (n-1)) * out.grad         out._backward = _backward        return outdef backward(node):    topo = []    visited = set()    def build_topo(v):        if v not in visited:            visited.add(v)            for child in v._prev:                build_topo(child)            topo.append(v)    build_topo(node)    node.grad = 1    for node in reversed(topo):        node._backward()# Example usagex = Node(2.0)y = x**4 + 3*x**2 + 2*xbackward(y)print(f\"f(2) = {y.value}, f'(2) = {x.grad}\")# Output: f(2) = 42.0, f'(2) = 58.0\n```\n\nThis implementation builds a computational graph and then traverses it backwards when the `backward` method is called on the output, to compute gradients, mimicking the process used in deep learning frameworks.\nThe **key difference** in computational complexity between **forward** and **reverse** modes becomes apparent when we consider functions with many inputs (parameters) and few outputs (typically a single loss value in ML), making reverse mode the preferred choice for deep learning [1]. the reason is that, in forward mode, computing the gradient for each input element xᵢ requires a **separate** forward pass through the computational graph.\n* * *\n## 4. Implementation Strategies: Operator Overloading vs. Source Transformation\n### 4.1 Operator Overloading\nOperator overloading, as demonstrated in our previous examples, redefines mathematical operations to compute both the result and its derivative. It's the approach used by PyTorch and many Python-based AD libraries [4].\n### 4.2 Source Transformation\nSource transformation analyzes and modifies the source code to insert derivative computations. While more complex to implement, it can lead to more optimized code, especially for **static** computational graphs [1]. Tools like Tapenade use this approach.\nHere's a conceptual example of how source transformation might work:\npython\n\n```\n# Original functiondef f(x):    return x**4 + 3*x**2 + 2*x# Transformed function (conceptual, not actual code)def f_and_gradient(x):    # Forward pass    t1 = x**2    t2 = t1**2    t3 = 3 * t1    t4 = 2 * x    y = t2 + t3 + t4    # Backward pass    dy = 1    dt4 = dy    dt3 = dy    dt2 = dy    dt1 = 2 * x * dt3 + 2 * t1 * dt2    dx = 4 * x**3 * dy + 6 * x * dy + 2 * dy    return y, dx\n```\n\nThis transformed version computes both the function value and its gradient in a single pass through the code. As you can see, it is not as flexible and scalable for large-scale purposes such as in deep learning.\n* * *\n## 5. AutoDiff in the Wild: PyTorch vs. JAX\n### 5.1 PyTorch\nPyTorch uses a dynamic computational graph, built on-the-fly as operations are performed. This allows for flexibility in network architecture and easier debugging [5].\npython\n\n```\nimport torchdef f(x):    return x**4 + 3*x**2 + 2*xx = torch.tensor([2.0], requires_grad=True)y = f(x)y.backward()print(f\"f(2) = {y.item()}, f'(2) = {x.grad.item()}\")# Output: f(2) = 42.0, f'(2) = 58.0\n```\n\nPyTorch's `autograd` engine records operations in a directed acyclic graph (DAG), where leaves are input tensors and roots are output tensors. During the backward pass, it computes gradients by traversing this graph [5].\nFor very detailed explanation to get a sense of how PyTorch's autograd works, i would extremely recommend the first to videos of Andrej Karpathy's [**Neural Networks: Zero to Hero**](https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ) playlist.\n### 5.2 JAX\nJAX, developed by Google Research, on the the hand uses a static computational graph and leverages XLA (Accelerated Linear Algebra) for efficient compilation to achieve better performance. It provides function transformations like `grad` for automatic differentiation, `vmap` for vectorization, and `jit` for compilation [4].\npython\n\n```\nimport jaximport jax.numpy as jnp@jax.jitdef f(x):    return x**4 + 3*x**2 + 2*xdf = jax.grad(f)x = 2.0print(f\"f(2) = {f(x)}, f'(2) = {df(x)}\")# Output: f(2) = 42.0, f'(2) = 58.0# Vectorized computationvdf = jax.vmap(df)x_vec = jnp.array([1.0, 2.0, 3.0])print(f\"f'(x) for x=[1,2,3]: {vdf(x_vec)}\")# Output: f'(x) for x=[1,2,3]: [10. 58. 154.]\n```\n\nJAX's approach allows for efficient compilation and execution on accelerators like GPUs and TPUs [4]. Check out the [**\"The Autodiff Cookbook\"**](https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html) from JAX developers for a more technical grasp of their implementations.\nNote the difference that PyTorch's implementation requires that first a forward pass is done with an input, then as the backwards are computed, the gradients are accessible, whereas in JAX, the `jax.grad` transformation can be called on any defined function without the need to calling the function itself manually.\n* * *\n## 6. Some Advanced Topics in AutoDiff\n### 6.1 Higher-Order Derivatives\nOne thing o note is that AutoDiff isn't limited to first-order derivatives. By applying AD to its own output, we can compute higher-order derivatives. This is crucial for optimization algorithms like Newton's method that use second-order information (Hessians).\nIn JAX particularly, computing higher-order derivatives is pretty straightforward:\npython\n\n```\nimport jaximport jax.numpy as jnpdef f(x):    return x**4 + 3*x**2 + 2*xddf = jax.grad(jax.grad(f))x = 2.0print(f\"f''(2) = {ddf(x)}\")# Output: f''(2) = 102.0\n```\n\njust call the `grad` function transformation twice on your function and you're good to go.\n### 6.2 Vector-Jacobian Products (VJPs) and Jacobian-Vector Products (JVPs)\nVJPs and JVPs are the building blocks of reverse and forward mode AD, respectively. Understanding these operations is crucial for implementing efficient custom gradients.\nJAX provides explicit functions for these operations:\npython\n\n```\nimport jaximport jax.numpy as jnpdef f(x):    return jnp.array([x**2, x**3])x = 2.0y, vjp_fn = jax.vjp(f, x)print(f\"VJP: {vjp_fn(jnp.array([1.0, 1.0]))[0]}\")primal, jvp_fn = jax.jvp(f, (x,), (1.0,))print(f\"JVP: {jvp_fn}\")# Output:# VJP: 16.0# JVP: [4. 12.]\n```\n\n### 6.3 AD through Iterative Processes\nApplying AD to iterative processes like optimization loops or recurrent neural networks requires careful handling to avoid excessive memory usage. Techniques like checkpointing and reversible computations are used to balance memory usage and computational cost [1].\n* * *\n## 7. The Impact of AutoDiff on Deep Learning\nAutoDiff, particularly **reverse** mode AD, has been instrumental in the deep learning revolution. It allows efficient computation of gradients for millions of parameters with respect to a loss value. This efficiency enables the training of increasingly complex models, driving advancements in areas like natural language processing, computer vision, and reinforcement learning [2].\nSome key impacts to mention:\n  1. **Architectural Flexibility** : AD allows researchers to easily experiment with novel network architectures without manually deriving gradients.\n  2. **Computational Efficiency** : Reverse mode AD makes it feasible to train very deep networks with millions or billions of parameters.\n  3. **Higher-Order Optimization** : Easy access to higher-order derivatives enables more sophisticated optimization techniques.\n  4. **Custom Differentiable Operations** : Researchers can define custom differentiable operations, expanding the range of possible model architectures.\n\n\n## 8. Conclusion and Future Directions\nAutomatic Differentiation has become an indispensable tool in machine learning, enabling the training of increasingly complex models. As we push the boundaries of AI, several exciting directions for AD research emerge:\n  1. **AD for Probabilistic Programming** : Extending AD to handle probabilistic computations and enable more flexible Bayesian inference.\n  2. **Differentiable Programming** : Moving beyond traditional neural networks to make entire programs differentiable.\n  3. **Hardware-Specific Optimizations** : Tailoring AD algorithms for specialized AI hardware.\n  4. **AD for Sparse and Structured Computations** : Developing efficient AD techniques for sparse or structured problems common in scientific computing.\n\n\nAs these areas develop, we can expect AutoDiff to continue playing a crucial role in advancing the field of machine learning and artificial intelligence.\n## References\n[1] Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2018). Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research, 18, 1–43.\n[2] Grosse, R. (2019). Automatic Differentiation. CSC421/2516 Lecture Notes, University of Toronto.\n[3] Andmholm. (2023). What is Automatic Differentiation. Hugging Face Blog.\n[4] JAX Team. (2024). Automatic Differentiation and the JAX Ecosystem. JAX Documentation.\n[5] PyTorch Team. (2024). Autograd: Automatic Differentiation. PyTorch Tutorials.\n[6] <https://simple-english-machine-learning.readthedocs.io/en/latest/neural-networks/computational-graphs.html>\n* * *\n![](https://assets.insightmediagroup.io/media/1787657518176_ovfxiv.jpeg)\nWritten By\n## [Ebrahim Pichka](https://towardsdatascience.com/author/ebrahimpichka/)\n[See all from Ebrahim Pichka](https://towardsdatascience.com/author/ebrahimpichka/)\n[Computer Science](https://towardsdatascience.com/tag/computer-science/)[Deep Learning](https://towardsdatascience.com/tag/deep-learning/)[Getting Started](https://towardsdatascience.com/tag/getting-started/)[Machine Learning](https://towardsdatascience.com/tag/machine-learning/)[Reinforcement Learning](https://towardsdatascience.com/tag/reinforcement-learning/)\nShare This Article\n  * [](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Ftowardsdatascience.com%2Fautomatic-differentiation-autodiff-a-brief-intro-with-examples-3f3d257ffe3b%2F&title=Automatic%20Differentiation%20\\(AutoDiff\\)%3A%20A%20Brief%20Intro%20with%20Examples)\n  * [](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Ftowardsdatascience.com%2Fautomatic-differentiation-autodiff-a-brief-intro-with-examples-3f3d257ffe3b%2F&title=Automatic%20Differentiation%20\\(AutoDiff\\)%3A%20A%20Brief%20Intro%20with%20Examples)\n  * [](https://x.com/share?url=https%3A%2F%2Ftowardsdatascience.com%2Fautomatic-differentiation-autodiff-a-brief-intro-with-examples-3f3d257ffe3b%2F&text=Automatic%20Differentiation%20\\(AutoDiff\\)%3A%20A%20Brief%20Intro%20with%20Examples)\n\n\nTowards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.\n[Write for TDS](https://towardsdatascience.com/submissions/)\n## Related Articles\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/0uknfAZETlIq_U3tz-scaled.jpg)](https://towardsdatascience.com/deep-learning-understand-the-principle-ad146d9f54dd/)\n## [Deep Learning: Understand the Principle](https://towardsdatascience.com/deep-learning-understand-the-principle-ad146d9f54dd/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA gentle introduction. What problem it solves and why the layered structure\n[Vinicius Monteiro](https://towardsdatascience.com/author/vinimo/)\nJanuary 11, 20214 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/03dU5rx6iYuAX44R7-scaled.jpg)](https://towardsdatascience.com/4-techniques-to-deal-with-missing-data-in-datasets-841f8a303395/)\n## [4 Techniques To Deal With Missing Data in Datasets](https://towardsdatascience.com/4-techniques-to-deal-with-missing-data-in-datasets-841f8a303395/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nSimple methods that can nullify the effects of missing values\n[Egor Howell](https://towardsdatascience.com/author/egorhowell/)\nSeptember 17, 20213 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/09/1OFWI0fLekGWWkn2w9kX7Aw.png)](https://towardsdatascience.com/here-comes-mamba-the-selective-state-space-model-435e5d17a451/)\n## [Here Comes Mamba: The Selective State Space Model](https://towardsdatascience.com/here-comes-mamba-the-selective-state-space-model-435e5d17a451/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nPart 3 - Towards Mamba State Space Models for Images, Videos and Time Series\n[Sascha Kirch](https://towardsdatascience.com/author/saschakirch/)\nSeptember 3, 202418 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1XSl5z9T7WCvf-7_wX6B0cA-scaled.jpeg)](https://towardsdatascience.com/4-must-know-objects-in-python-datetime-module-d18565fddd05/)\n## [4 Must-Know Objects in Python Datetime Module](https://towardsdatascience.com/4-must-know-objects-in-python-datetime-module-d18565fddd05/)\n[Artificial Intelligence](https://towardsdatascience.com/category/artificial-intelligence/)\nA comprehensive practical guide\n[Soner Yıldırım](https://towardsdatascience.com/author/sonery/)\nApril 13, 20214 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1kigPUGyriodovbOTcxdEhw.jpeg)](https://towardsdatascience.com/what-you-need-to-know-before-taking-the-machine-learning-course-by-stanford-84fd7bf94628/)\n## [What you need to know before taking the Machine Learning course by Stanford](https://towardsdatascience.com/what-you-need-to-know-before-taking-the-machine-learning-course-by-stanford-84fd7bf94628/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nMachine Learning course by Andrew Ng & Stanford review\n[Dmytro Nikolaiev (Dimid)](https://towardsdatascience.com/author/andimid/)\nOctober 29, 20215 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1O6PJn8L76ejOrixwDPNFGg.png)](https://towardsdatascience.com/coin-counting-using-lang-sam-b469827808a7/)\n## [Coin Counting using Lang-SAM](https://towardsdatascience.com/coin-counting-using-lang-sam-b469827808a7/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nLeverage Lang-SAM for precise coin segmentation in images, followed by a custom neural network to identify and sum their monetary values.\n[Gamze Zorlubas](https://towardsdatascience.com/author/gamzsaglam/)\nOctober 3, 202310 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/0ngZbj1DuUHq2SmrY.jpg)](https://towardsdatascience.com/the-math-behind-batch-normalization-90ebbc0b1b0b/)\n## [The Math Behind Batch Normalization](https://towardsdatascience.com/the-math-behind-batch-normalization-90ebbc0b1b0b/)\n[Deep Learning](https://towardsdatascience.com/category/artificial-intelligence/deep-learning/)\nExplore Batch Normalization, a cornerstone of neural networks, understand its mathematics and implement it from scratch.\n[Cristian Leo](https://towardsdatascience.com/author/cristianleo120/)\nMay 8, 202423 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/09e-O5bruClQjVVQh-scaled.jpg)](https://towardsdatascience.com/build-a-flask-heroku-mood-tracker-web-app-using-the-spotify-api-14b3b5c92ac9/)\n## [Build a Flask-Heroku Mood Tracker Web App Using the Spotify API](https://towardsdatascience.com/build-a-flask-heroku-mood-tracker-web-app-using-the-spotify-api-14b3b5c92ac9/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nA-Z project to generate mood scores based on the songs I listen to everyday and color my online mood tracker accordingly\n[Irene Chang](https://towardsdatascience.com/author/irenechang1510/)\nNovember 9, 20213 min read\n  * [![](https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/03NjSza4OqlpPr7_d.jpg)](https://towardsdatascience.com/mean-average-precision-at-k-map-k-clearly-explained-538d8e032d2/)\n## [Mean Average Precision at K (MAP@K) clearly explained](https://towardsdatascience.com/mean-average-precision-at-k-map-k-clearly-explained-538d8e032d2/)\n[Machine Learning](https://towardsdatascience.com/category/artificial-intelligence/machine-learning/)\nOne of the most popular evaluation metrics for recommender or ranking problems step by step explained\n[Konstantin Rink](https://towardsdatascience.com/author/konstantin-rink/)\nJanuary 18, 20237 min read\n\n\n  * [YouTube](https://www.youtube.com/c/TowardsDataScience)\n  * [X](https://x.com/TDataScience)\n  * [LinkedIn](https://www.linkedin.com/company/towards-data-science/)\n  * [Threads](https://www.threads.net/@towardsdatascience)\n  * [Bluesky](https://bsky.app/profile/towardsdatascience.com)\n\n\n[](https://towardsdatascience.com/)\nYour home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.\nSubscribe to Our Newsletter[Write For TDS](https://towardsdatascience.com/submissions/)•[About](https://towardsdatascience.com/about-towards-data-science-d691af11cc2f/)•[Advertise](https://contact.towardsdatascience.com/advertise-with-towards-data-science)•[Privacy Policy](https://towardsdatascience.com/privacy-policy/)•[Terms of Use](https://towardsdatascience.com/website-terms-of-use/)\n© Insight Media Group, LLC 2026•[The New Stack](https://thenewstack.io)•[roadmap.sh](https://roadmap.sh)•[Creator Network](https://insightmediagroup.io/creator-network)\n",
  "metadata": {
    "title": "Automatic Differentiation (AutoDiff): A Brief Intro with Examples | Towards Data Science",
    "description": "An introduction to the mechanics of AutoDiff, exploring its mathematical principles, implementation strategies, and applications",
    "keywords": null,
    "author": "Ebrahim Pichka",
    "og:locale": "en_US",
    "og:type": "article",
    "og:title": "Automatic Differentiation (AutoDiff): A Brief Intro with Examples",
    "og:description": "An introduction to the mechanics of AutoDiff, exploring its mathematical principles, implementation strategies, and applications",
    "og:url": "https://towardsdatascience.com/automatic-differentiation-autodiff-a-brief-intro-with-examples-3f3d257ffe3b/",
    "og:site_name": "Towards Data Science",
    "og:image": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/02zoQV7HydfU2dV8c-scaled.jpg",
    "og:image:width": "2560",
    "og:image:height": "1702",
    "og:image:type": "image/jpeg",
    "twitter:card": "summary_large_image",
    "twitter:site": "@TDataScience",
    "twitter:creator": "@TDataScience",
    "twitter:label1": "Written by",
    "twitter:data1": "Ebrahim Pichka",
    "twitter:label2": "Est. reading time",
    "twitter:data2": "11 min read",
    "article:published_time": "2024-10-11T19:07:55.000Z",
    "article:modified_time": "2024-10-11T19:07:55.000Z",
    "article:author": "https://towardsdatascience.com/author/ebrahimpichka/"
  },
  "paywall": {
    "listed_paid": true,
    "detected": false,
    "marker": "",
    "public_only": true,
    "policy": "付费/会员内容只保留页面公开可见部分，不绕过付费墙"
  },
  "crawl": {
    "success": true,
    "status_code": 200,
    "error": ""
  },
  "code_blocks": [
    {
      "language": "txt",
      "file": "attachments/dceb332dcd69/code/block_000.txt",
      "chars": 724
    },
    {
      "language": "txt",
      "file": "attachments/dceb332dcd69/code/block_001.txt",
      "chars": 1559
    },
    {
      "language": "txt",
      "file": "attachments/dceb332dcd69/code/block_002.txt",
      "chars": 388
    },
    {
      "language": "txt",
      "file": "attachments/dceb332dcd69/code/block_003.txt",
      "chars": 202
    },
    {
      "language": "txt",
      "file": "attachments/dceb332dcd69/code/block_004.txt",
      "chars": 343
    },
    {
      "language": "txt",
      "file": "attachments/dceb332dcd69/code/block_005.txt",
      "chars": 158
    },
    {
      "language": "txt",
      "file": "attachments/dceb332dcd69/code/block_006.txt",
      "chars": 257
    }
  ],
  "images": [
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/10/02zoQV7HydfU2dV8c-scaled.jpg",
      "alt": "Photo by Bozhin Karaivanov on Unsplash",
      "local_path": "attachments/dceb332dcd69/images/000.jpg",
      "bytes": 251935,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/01/0uknfAZETlIq_U3tz-scaled.jpg",
      "alt": "",
      "local_path": "attachments/dceb332dcd69/images/001.jpg",
      "bytes": 406449,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/09/03dU5rx6iYuAX44R7-scaled.jpg",
      "alt": "",
      "local_path": "attachments/dceb332dcd69/images/002.jpg",
      "bytes": 219922,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/09/1OFWI0fLekGWWkn2w9kX7Aw.png",
      "alt": "",
      "local_path": "attachments/dceb332dcd69/images/003.png",
      "bytes": 2935816,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/04/1XSl5z9T7WCvf-7_wX6B0cA-scaled.jpeg",
      "alt": "",
      "local_path": "attachments/dceb332dcd69/images/004.jpeg",
      "bytes": 165032,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/10/1kigPUGyriodovbOTcxdEhw.jpeg",
      "alt": "",
      "local_path": "attachments/dceb332dcd69/images/005.jpeg",
      "bytes": 234708,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/10/1O6PJn8L76ejOrixwDPNFGg.png",
      "alt": "",
      "local_path": "attachments/dceb332dcd69/images/006.png",
      "bytes": 466690,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2024/05/0ngZbj1DuUHq2SmrY.jpg",
      "alt": "",
      "local_path": "attachments/dceb332dcd69/images/007.jpg",
      "bytes": 379588,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2021/11/09e-O5bruClQjVVQh-scaled.jpg",
      "alt": "",
      "local_path": "attachments/dceb332dcd69/images/008.jpg",
      "bytes": 333547,
      "downloaded": true
    },
    {
      "src": "https://assets.insightmediagroup.io/media/wp-content/uploads/2023/01/03NjSza4OqlpPr7_d.jpg",
      "alt": "",
      "local_path": "attachments/dceb332dcd69/images/009.jpg",
      "bytes": 159874,
      "downloaded": true
    }
  ],
  "audio": [],
  "videos": [
    {
      "src": "https://youtu.be/wG_nF1awSSY?t=305",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    },
    {
      "src": "https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ",
      "quality": "unavailable",
      "local_path": "",
      "note": "无1080p且无720p，未交付更低清晰度"
    }
  ],
  "links": [
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/latest/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/editors-pick/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/the-variable/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/shipai/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/submissions/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/ebrahimpichka/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/computer-science/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/deep-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/getting-started/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/tag/reinforcement-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/deep-learning-understand-the-principle-ad146d9f54dd/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/vinimo/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/4-techniques-to-deal-with-missing-data-in-datasets-841f8a303395/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/category/artificial-intelligence/machine-learning/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/author/egorhowell/"
    },
    {
      "type": "internal",
      "href": "https://towardsdatascience.com/here-comes-mamba-the-selective-state-space-model-435e5d17a451/"
    },
    {
      "type": "external",
      "href": "https://write.insightmediagroup.io"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/company/towards-data-science/"
    },
    {
      "type": "external",
      "href": "https://x.com/TDataScience"
    },
    {
      "type": "external",
      "href": "https://unsplash.com/@bkaraivanov"
    },
    {
      "type": "external",
      "href": "https://unsplash.com"
    },
    {
      "type": "external",
      "href": "https://roadmap.sh/machine-learning"
    },
    {
      "type": "external",
      "href": "https://tutorial.math.lamar.edu/classes/calci/chainrule.aspx"
    },
    {
      "type": "external",
      "href": "https://maximerobeyns.com/of_vjps_and_jvps"
    },
    {
      "type": "external",
      "href": "https://youtu.be/wG_nF1awSSY?t=305"
    },
    {
      "type": "external",
      "href": "https://simple-english-machine-learning.readthedocs.io/en/latest/neural-networks/computational-graphs.html"
    },
    {
      "type": "external",
      "href": "https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ"
    },
    {
      "type": "external",
      "href": "https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html"
    },
    {
      "type": "external",
      "href": "https://www.facebook.com/sharer/sharer.php?title=Automatic+Differentiation+%28AutoDiff%29%3A+A+Brief+Intro+with+Examples&u=https%3A%2F%2Ftowardsdatascience.com%2Fautomatic-differentiation-autodiff-a-brief-intro-with-examples-3f3d257ffe3b%2F"
    },
    {
      "type": "external",
      "href": "https://www.linkedin.com/shareArticle?mini=true&title=Automatic+Differentiation+%28AutoDiff%29%3A+A+Brief+Intro+with+Examples&url=https%3A%2F%2Ftowardsdatascience.com%2Fautomatic-differentiation-autodiff-a-brief-intro-with-examples-3f3d257ffe3b%2F"
    },
    {
      "type": "external",
      "href": "https://x.com/share?text=Automatic+Differentiation+%28AutoDiff%29%3A+A+Brief+Intro+with+Examples&url=https%3A%2F%2Ftowardsdatascience.com%2Fautomatic-differentiation-autodiff-a-brief-intro-with-examples-3f3d257ffe3b%2F"
    },
    {
      "type": "external",
      "href": "https://www.youtube.com/c/TowardsDataScience"
    },
    {
      "type": "external",
      "href": "https://www.threads.net/@towardsdatascience"
    },
    {
      "type": "external",
      "href": "https://bsky.app/profile/towardsdatascience.com"
    },
    {
      "type": "external",
      "href": "https://thenewstack.io"
    }
  ],
  "repair_version": 2,
  "repaired_at": "2026-10-14T11:11:49Z",
  "source_url": "https://towardsdatascience.com/automatic-differentiation-autodiff-a-brief-intro-with-examples-3f3d257ffe3b/",
  "delivery_status": "excluded_paid",
  "discovery_status": "excluded_paid",
  "missing_reason": "按原约定排除付费/会员全文"
}